[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:18 dist-test-slave-jpj1 krb5kdc[23824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:18 dist-test-slave-jpj1 krb5kdc[23824](info): set up 2 sockets
Aug 10 18:11:18 dist-test-slave-jpj1 krb5kdc[23824](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 18:11:20.264309 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.441s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260810 18:11:20.437477 23813 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.173s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260810 18:11:20.574363 23813 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.137s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 18:11:20.707056 23813 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.132s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:20 dist-test-slave-jpj1 krb5kdc[23824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:20.806854 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:37567
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:37567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:20.923031 23848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:20.923297 23848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:20.923336 23848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:20.927457 23848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:20.927524 23848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:20.927572 23848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:20.927627 23848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:20.927652 23848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:20.933365 23848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:37567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:20.934835 23848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:20.936165 23848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:20.950011 23856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:20.950204 23858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:20.950475 23854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:20.950700 23848 server_base.cc:1034] running on GCE node
I20260810 18:11:20.951491 23848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:20.953264 23848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:20.954433 23848 hybrid_clock.cc:648] HybridClock initialized: now 1786385480954412 us; error 31 us; skew 500 ppm
Aug 10 18:11:20 dist-test-slave-jpj1 krb5kdc[23824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385480, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:20.957412 23848 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:20.958628 23848 webserver.cc:460] Webserver started at http://127.23.65.126:39739/ using document root <none> and password file <none>
I20260810 18:11:20.959509 23848 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:20.959631 23848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:20.960057 23848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:20.962920 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "7784d662c21b4cfd9e8a1719e7f988cb"
format_stamp: "Formatted at 2026-08-10 18:11:20 on dist-test-slave-jpj1"
server_key: "e2a14fc1e1300154c3c1a8a11c9b5732"
server_key_iv: "f7e6a0e11b3b3ac4e1eedb316c9cb0a4"
server_key_version: "encryptionkey@0"
I20260810 18:11:20.963846 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "7784d662c21b4cfd9e8a1719e7f988cb"
format_stamp: "Formatted at 2026-08-10 18:11:20 on dist-test-slave-jpj1"
server_key: "e2a14fc1e1300154c3c1a8a11c9b5732"
server_key_iv: "f7e6a0e11b3b3ac4e1eedb316c9cb0a4"
server_key_version: "encryptionkey@0"
I20260810 18:11:20.969023 23848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 18:11:20.972635 23872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:20.974253 23848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:20.974439 23848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "7784d662c21b4cfd9e8a1719e7f988cb"
format_stamp: "Formatted at 2026-08-10 18:11:20 on dist-test-slave-jpj1"
server_key: "e2a14fc1e1300154c3c1a8a11c9b5732"
server_key_iv: "f7e6a0e11b3b3ac4e1eedb316c9cb0a4"
server_key_version: "encryptionkey@0"
I20260810 18:11:20.974572 23848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:20.986697 23848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:20.990233 23848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:20.990453 23848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:21.002461 23848 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:37567
I20260810 18:11:21.002501 23956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:37567 every 8 connection(s)
I20260810 18:11:21.004045 23848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:21.009423 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:21.013535 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 23848
I20260810 18:11:21.013671 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:21.015079 23813 external_mini_cluster.cc:1428] Setting key c88b65ebcb1a2b7ee9eb828b36b17d18
I20260810 18:11:21.019348 23957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb: Bootstrap starting.
I20260810 18:11:21.024488 23957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:21.026166 23957 log.cc:824] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:21.029186 23957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb: No bootstrap required, opened a new log
I20260810 18:11:21.036881 23957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7784d662c21b4cfd9e8a1719e7f988cb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37567 } }
I20260810 18:11:21.037292 23957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:21.037395 23957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7784d662c21b4cfd9e8a1719e7f988cb, State: Initialized, Role: FOLLOWER
I20260810 18:11:21.038414 23957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7784d662c21b4cfd9e8a1719e7f988cb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37567 } }
I20260810 18:11:21.038614 23957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:21.038757 23957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:21.038893 23957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:21.040354 23957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7784d662c21b4cfd9e8a1719e7f988cb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37567 } }
Aug 10 18:11:21 dist-test-slave-jpj1 krb5kdc[23824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:21.040992 23957 leader_election.cc:302] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7784d662c21b4cfd9e8a1719e7f988cb; no voters:
I20260810 18:11:21.041379 23957 leader_election.cc:288] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:21.041540 23963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:21.041913 23963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [term 1 LEADER]: Becoming Leader. State: Replica: 7784d662c21b4cfd9e8a1719e7f988cb, State: Running, Role: LEADER
I20260810 18:11:21.042366 23963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7784d662c21b4cfd9e8a1719e7f988cb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37567 } }
I20260810 18:11:21.043244 23957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:21.045643 23964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7784d662c21b4cfd9e8a1719e7f988cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7784d662c21b4cfd9e8a1719e7f988cb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37567 } } }
I20260810 18:11:21.045771 23964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:21.046106 23965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7784d662c21b4cfd9e8a1719e7f988cb. Latest consensus state: current_term: 1 leader_uuid: "7784d662c21b4cfd9e8a1719e7f988cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7784d662c21b4cfd9e8a1719e7f988cb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37567 } } }
I20260810 18:11:21.046329 23965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:21.046674 23978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:21.049185 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:21.020004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.23.65.126:37567)
0810 18:11:21.020539 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:21.020560 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:21.024149 (+ 3589us) server_negotiation.cc:407] Connection header received
0810 18:11:21.028845 (+ 4696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:21.028870 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:21.029374 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:21.029762 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:21.032194 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.034317 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.035789 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.036618 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.036821 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:21.041380 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:21.041425 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:21.041443 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:21.041474 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:21.043661 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:21.044319 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:21.044326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:21.044332 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:21.044486 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:21.044849 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:21.044852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:21.044854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:21.045497 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:21.045695 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:21.047697 (+ 2002us) server_negotiation.cc:299] Negotiation successful
0810 18:11:21.048000 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":129,"threads_started":1}
I20260810 18:11:21.052886 23978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:21.060992 23978 catalog_manager.cc:1347] Generated new cluster ID: af23a30e075f4dcca704e985dab46887
I20260810 18:11:21.061072 23978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:21.081655 23978 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:21.082988 23978 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:21.090773 23978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7784d662c21b4cfd9e8a1719e7f988cb: Generated new TSK 0
I20260810 18:11:21.091854 23978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:11:21.134150 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37567
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:21.308514 24004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:21.308800 24004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:21.308867 24004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:21.312883 24004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:21.312943 24004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:21.313024 24004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:21.318544 24004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:21.319705 24004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:21.320801 24004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:21.333324 24014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:21.333355 24010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:21.333523 24004 server_base.cc:1034] running on GCE node
W20260810 18:11:21.333355 24011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:21.333976 24004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:21.334580 24004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:21.335790 24004 hybrid_clock.cc:648] HybridClock initialized: now 1786385481335761 us; error 40 us; skew 500 ppm
Aug 10 18:11:21 dist-test-slave-jpj1 krb5kdc[23824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385481, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:21.339368 24004 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:11:21.340257 24004 webserver.cc:460] Webserver started at http://127.23.65.65:35561/ using document root <none> and password file <none>
I20260810 18:11:21.340775 24004 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:21.340833 24004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:21.340993 24004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:21.343070 24004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "0517dcee99e44d1e91cacf8fe15d0c06"
format_stamp: "Formatted at 2026-08-10 18:11:21 on dist-test-slave-jpj1"
server_key: "8a83b755d1af7278792a6cb93cb2ad04"
server_key_iv: "c3d5480bd27af35e851672df1092fe34"
server_key_version: "encryptionkey@0"
I20260810 18:11:21.343688 24004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "0517dcee99e44d1e91cacf8fe15d0c06"
format_stamp: "Formatted at 2026-08-10 18:11:21 on dist-test-slave-jpj1"
server_key: "8a83b755d1af7278792a6cb93cb2ad04"
server_key_iv: "c3d5480bd27af35e851672df1092fe34"
server_key_version: "encryptionkey@0"
I20260810 18:11:21.347962 24004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:11:21.350478 24027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:21.351732 24004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:11:21.351828 24004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "0517dcee99e44d1e91cacf8fe15d0c06"
format_stamp: "Formatted at 2026-08-10 18:11:21 on dist-test-slave-jpj1"
server_key: "8a83b755d1af7278792a6cb93cb2ad04"
server_key_iv: "c3d5480bd27af35e851672df1092fe34"
server_key_version: "encryptionkey@0"
I20260810 18:11:21.351912 24004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:21.375663 24004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:21.379081 24004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:21.379338 24004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:21.380273 24004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:21.381507 24004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:21.381570 24004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:21.381629 24004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:21.381662 24004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:21.402201 24004 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:38675
I20260810 18:11:21.402516 24201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:38675 every 8 connection(s)
I20260810 18:11:21.403287 24004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:21.413506 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24004
I20260810 18:11:21.413622 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:21.413921 23813 external_mini_cluster.cc:1428] Setting key a0a99d7ffb855852530046931698872e
Aug 10 18:11:21 dist-test-slave-jpj1 krb5kdc[23824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385481, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:21.419517 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:21.406205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:51877 (local address 127.23.65.126:37567)
0810 18:11:21.406351 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:21.406356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:21.407489 (+ 1133us) server_negotiation.cc:407] Connection header received
0810 18:11:21.409653 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:21.409658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:21.409722 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:21.409823 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:21.411215 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.411822 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.413019 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.413234 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.413300 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:21.416312 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:21.416316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:21.416318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:21.416342 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:21.417583 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:21.418070 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:21.418073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:21.418075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:21.418156 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:21.418469 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:21.418473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:21.418474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:21.418660 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:21.418747 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:21.419228 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 18:11:21.419354 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 18:11:21.420341 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:21.406622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:37567 (local address 127.23.65.65:51877)
0810 18:11:21.407325 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:21.407385 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:21.409412 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:21.409991 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:21.410000 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:21.410446 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:21.411076 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:21.411093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.411919 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.411921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:21.412919 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:21.412929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.413363 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.413367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:21.413651 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:21.414370 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:21.414421 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:21.416158 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:21.417708 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:21.417714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:21.417729 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:21.417976 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:21.418238 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:21.418241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:21.418243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:21.418382 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:21.418762 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:21.418768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:21.418993 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 18:11:21.419159 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 18:11:21.419470 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":157,"threads_started":1}
I20260810 18:11:21.421566 24202 heartbeater.cc:347] Connected to a master server at 127.23.65.126:37567
I20260810 18:11:21.421821 24202 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:21.422362 24202 heartbeater.cc:511] Master 127.23.65.126:37567 requested a full tablet report, sending...
I20260810 18:11:21.424801 23903 ts_manager.cc:194] Registered new tserver with Master: 0517dcee99e44d1e91cacf8fe15d0c06 (127.23.65.65:38675)
I20260810 18:11:21.426878 23903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:51877
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:11:21.446393 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37567
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:21.596195 24210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:21.596449 24210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:21.596511 24210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:21.600459 24210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:21.600514 24210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:21.600589 24210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:21.605970 24210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:21.606976 24210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:21.608027 24210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:21.617928 24220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:21.617972 24222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:21.618024 24219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:21.618325 24210 server_base.cc:1034] running on GCE node
I20260810 18:11:21.618716 24210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:21.619310 24210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:21.620464 24210 hybrid_clock.cc:648] HybridClock initialized: now 1786385481620433 us; error 48 us; skew 500 ppm
Aug 10 18:11:21 dist-test-slave-jpj1 krb5kdc[23824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385481, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:21.623649 24210 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:11:21.624434 24210 webserver.cc:460] Webserver started at http://127.23.65.66:39027/ using document root <none> and password file <none>
I20260810 18:11:21.624925 24210 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:21.624981 24210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:21.625145 24210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:21.627265 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "e7ffef84a70d40ffb798888039c39480"
format_stamp: "Formatted at 2026-08-10 18:11:21 on dist-test-slave-jpj1"
server_key: "b7b55bd9681f919896202b0214b798ba"
server_key_iv: "33aca130f99e7618846752cee1dd3247"
server_key_version: "encryptionkey@0"
I20260810 18:11:21.627871 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "e7ffef84a70d40ffb798888039c39480"
format_stamp: "Formatted at 2026-08-10 18:11:21 on dist-test-slave-jpj1"
server_key: "b7b55bd9681f919896202b0214b798ba"
server_key_iv: "33aca130f99e7618846752cee1dd3247"
server_key_version: "encryptionkey@0"
I20260810 18:11:21.631611 24210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:11:21.633985 24235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:21.635262 24210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:21.635355 24210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "e7ffef84a70d40ffb798888039c39480"
format_stamp: "Formatted at 2026-08-10 18:11:21 on dist-test-slave-jpj1"
server_key: "b7b55bd9681f919896202b0214b798ba"
server_key_iv: "33aca130f99e7618846752cee1dd3247"
server_key_version: "encryptionkey@0"
I20260810 18:11:21.635437 24210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:21.653077 24210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:21.656061 24210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:21.656265 24210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:21.657076 24210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:21.657969 24210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:21.658018 24210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:21.658057 24210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:21.658077 24210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:21.680140 24210 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:33737
I20260810 18:11:21.680269 24429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:33737 every 8 connection(s)
I20260810 18:11:21.681320 24210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:21.683851 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24210
I20260810 18:11:21.683939 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:21.684176 23813 external_mini_cluster.cc:1428] Setting key 9d9f71f34235bbb2bc0a01283e9db290
Aug 10 18:11:21 dist-test-slave-jpj1 krb5kdc[23824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385481, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:11:21.696616 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:21.683466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:38963 (local address 127.23.65.126:37567)
0810 18:11:21.683660 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:21.683665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:21.684851 (+ 1186us) server_negotiation.cc:407] Connection header received
0810 18:11:21.686549 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:21.686553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:21.686616 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:21.686713 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:21.688293 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.688835 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.690229 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.690457 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.690546 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:21.693302 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:21.693305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:21.693307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:21.693332 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:21.694617 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:21.695111 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:21.695114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:21.695115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:21.695195 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:21.695553 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:21.695557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:21.695559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:21.695750 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:21.695840 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:21.696328 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 18:11:21.696449 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 18:11:21.697443 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:21.683938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:37567 (local address 127.23.65.66:38963)
0810 18:11:21.684685 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:21.684743 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:21.686322 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:21.686859 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:21.686866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:21.687410 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:21.688134 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:21.688152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.688965 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.688968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:21.690101 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:21.690112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.690598 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.690603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:21.690840 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:21.691512 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:21.691529 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:21.693139 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:21.694724 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:21.694731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:21.694745 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:21.694992 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:21.695278 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:21.695282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:21.695284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:21.695422 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:21.695851 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:21.695857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:21.696088 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 18:11:21.696250 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 18:11:21.696579 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":273,"threads_started":1}
I20260810 18:11:21.698576 24431 heartbeater.cc:347] Connected to a master server at 127.23.65.126:37567
I20260810 18:11:21.698834 24431 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
I20260810 18:11:21.699397 24431 heartbeater.cc:511] Master 127.23.65.126:37567 requested a full tablet report, sending...
I20260810 18:11:21.700455 23903 ts_manager.cc:194] Registered new tserver with Master: e7ffef84a70d40ffb798888039c39480 (127.23.65.66:33737)
I20260810 18:11:21.701012 23903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:38963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:11:21.717017 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37567
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:21.887642 24436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:21.887887 24436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:21.887933 24436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:21.891963 24436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:21.892020 24436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:21.892094 24436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:21.897583 24436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:21.898587 24436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:21.899602 24436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:21.910670 24449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:21.910765 24448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:21.910580 24458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:21.911244 24436 server_base.cc:1034] running on GCE node
I20260810 18:11:21.911661 24436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:21.912266 24436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:21.913421 24436 hybrid_clock.cc:648] HybridClock initialized: now 1786385481913400 us; error 34 us; skew 500 ppm
Aug 10 18:11:21 dist-test-slave-jpj1 krb5kdc[23824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385481, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:21.916843 24436 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:11:21.917757 24436 webserver.cc:460] Webserver started at http://127.23.65.67:45561/ using document root <none> and password file <none>
I20260810 18:11:21.918282 24436 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:21.918347 24436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:21.918543 24436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:21.920718 24436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "dc1f7d7b2ecd418b860f3fadb4406b61"
format_stamp: "Formatted at 2026-08-10 18:11:21 on dist-test-slave-jpj1"
server_key: "0876bbf5e66ace329c6d4b56c574e79e"
server_key_iv: "9bd731a39b4e229cb8142020221b7455"
server_key_version: "encryptionkey@0"
I20260810 18:11:21.921343 24436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "dc1f7d7b2ecd418b860f3fadb4406b61"
format_stamp: "Formatted at 2026-08-10 18:11:21 on dist-test-slave-jpj1"
server_key: "0876bbf5e66ace329c6d4b56c574e79e"
server_key_iv: "9bd731a39b4e229cb8142020221b7455"
server_key_version: "encryptionkey@0"
I20260810 18:11:21.925307 24436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:11:21.927889 24467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:21.929133 24436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:11:21.929262 24436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "dc1f7d7b2ecd418b860f3fadb4406b61"
format_stamp: "Formatted at 2026-08-10 18:11:21 on dist-test-slave-jpj1"
server_key: "0876bbf5e66ace329c6d4b56c574e79e"
server_key_iv: "9bd731a39b4e229cb8142020221b7455"
server_key_version: "encryptionkey@0"
I20260810 18:11:21.929358 24436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:21.957216 24436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:21.959936 24436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:21.960091 24436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:21.960789 24436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:21.961853 24436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:21.961900 24436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:21.961941 24436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:21.961959 24436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:21.989923 24436 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:43501
I20260810 18:11:21.990115 24663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:43501 every 8 connection(s)
I20260810 18:11:21.991092 24436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:21.996632 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24436
I20260810 18:11:21.996778 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:21.997092 23813 external_mini_cluster.cc:1428] Setting key 225c91dfcc40e418b647617cef5ecdb4
Aug 10 18:11:21 dist-test-slave-jpj1 krb5kdc[23824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385481, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:22.007078 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:21.993187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:41335 (local address 127.23.65.126:37567)
0810 18:11:21.993352 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:21.993357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:21.994562 (+ 1205us) server_negotiation.cc:407] Connection header received
0810 18:11:21.996212 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:21.996216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:21.996265 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:21.996344 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:21.998275 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.998908 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:22.000325 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:22.000541 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:22.000630 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:22.003454 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:22.003456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:22.003458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:22.003484 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:22.004970 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:22.005567 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:22.005569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:22.005570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:22.005646 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:22.005998 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:22.006000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:22.006001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:22.006169 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:22.006252 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:22.006845 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 18:11:22.006958 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:11:22.008251 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:21.993676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:37567 (local address 127.23.65.67:41335)
0810 18:11:21.994413 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:21.994460 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:21.995995 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:21.996560 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:21.996569 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:21.997333 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:21.998123 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:21.998143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:21.999036 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:21.999039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:22.000208 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:22.000219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:22.000681 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:22.000685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:22.000926 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:22.001546 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:22.001564 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:22.003299 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:22.005114 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:22.005120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:22.005135 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:22.005478 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:22.005745 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:22.005748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:22.005750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:22.005924 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:22.006261 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:22.006267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:22.006563 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 18:11:22.006778 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 18:11:22.007113 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":201,"threads_started":1}
I20260810 18:11:22.009526 24664 heartbeater.cc:347] Connected to a master server at 127.23.65.126:37567
I20260810 18:11:22.009778 24664 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:22.010305 24664 heartbeater.cc:511] Master 127.23.65.126:37567 requested a full tablet report, sending...
I20260810 18:11:22.011377 23903 ts_manager.cc:194] Registered new tserver with Master: dc1f7d7b2ecd418b860f3fadb4406b61 (127.23.65.67:43501)
I20260810 18:11:22.011924 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:11:22.012279 23903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:41335
Aug 10 18:11:22 dist-test-slave-jpj1 krb5kdc[23824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:11:22 dist-test-slave-jpj1 krb5kdc[23824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:22.045708 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:22.036220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.23.65.65:38675)
0810 18:11:22.036572 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:22.036579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:22.036605 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:11:22.036715 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:22.036721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:22.036917 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:22.037044 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:22.037745 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:22.038500 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:22.039360 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:22.039806 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:22.039896 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:22.042306 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:22.042322 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:22.042336 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:22.042363 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:22.044025 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:22.044418 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:22.044423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:22.044424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:22.044505 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:22.044698 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:22.044701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:22.044702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:22.044978 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:22.045141 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:22.045356 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 18:11:22.045512 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":140,"threads_started":1}
W20260810 18:11:22.046965 24072 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45090
Aug 10 18:11:22 dist-test-slave-jpj1 krb5kdc[23824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:11:22 dist-test-slave-jpj1 krb5kdc[23824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:22.079170 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:22.071145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.23.65.65:38675)
0810 18:11:22.071328 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:22.071332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:22.071569 (+ 237us) server_negotiation.cc:407] Connection header received
0810 18:11:22.071749 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:22.071752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:22.071798 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:22.071880 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:22.072519 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:22.073059 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:22.073885 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:22.074108 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:22.074187 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:22.076582 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:22.076586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:22.076588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:22.076615 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:22.077802 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:22.078162 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:22.078165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:22.078166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:22.078236 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:22.078418 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:22.078419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:22.078420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:22.078573 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:22.078671 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:22.078906 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 18:11:22.079012 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 18:11:22.080464 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24004
I20260810 18:11:22.094239 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24210
I20260810 18:11:22.105716 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24436
I20260810 18:11:22.123570 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 23848
2026-08-10T18:11:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3344 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:22 dist-test-slave-jpj1 krb5kdc[24685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:22 dist-test-slave-jpj1 krb5kdc[24685](info): set up 2 sockets
Aug 10 18:11:22 dist-test-slave-jpj1 krb5kdc[24685](info): commencing operation
krb5kdc: starting...
W20260810 18:11:23.289314 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:23 dist-test-slave-jpj1 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:23.410676 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40023
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:23.520107 24706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:23.520363 24706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:23.520403 24706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:23.524322 24706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:23.524374 24706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:23.524391 24706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:23.524407 24706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:23.524420 24706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:23.529626 24706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:40023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:23.530563 24706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:23.531486 24706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:23.539511 24716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:23.539562 24713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:23.539561 24714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:23.539770 24706 server_base.cc:1034] running on GCE node
I20260810 18:11:23.540392 24706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:23.541364 24706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:23.542520 24706 hybrid_clock.cc:648] HybridClock initialized: now 1786385483542504 us; error 26 us; skew 500 ppm
Aug 10 18:11:23 dist-test-slave-jpj1 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385483, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:23.545094 24706 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:23.545935 24706 webserver.cc:460] Webserver started at http://127.23.65.126:43731/ using document root <none> and password file <none>
I20260810 18:11:23.546427 24706 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:23.546482 24706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:23.546643 24706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:23.548588 24706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "5cf78dd4f50843ea9165cbe76cf79631"
format_stamp: "Formatted at 2026-08-10 18:11:23 on dist-test-slave-jpj1"
server_key: "d211ffa24dcd8c5c36a6037bf5ce85af"
server_key_iv: "92062ee866d37246a0b309a85c63ee85"
server_key_version: "encryptionkey@0"
I20260810 18:11:23.549077 24706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "5cf78dd4f50843ea9165cbe76cf79631"
format_stamp: "Formatted at 2026-08-10 18:11:23 on dist-test-slave-jpj1"
server_key: "d211ffa24dcd8c5c36a6037bf5ce85af"
server_key_iv: "92062ee866d37246a0b309a85c63ee85"
server_key_version: "encryptionkey@0"
I20260810 18:11:23.552686 24706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 18:11:23.554879 24728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:23.556032 24706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:11:23.556138 24706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "5cf78dd4f50843ea9165cbe76cf79631"
format_stamp: "Formatted at 2026-08-10 18:11:23 on dist-test-slave-jpj1"
server_key: "d211ffa24dcd8c5c36a6037bf5ce85af"
server_key_iv: "92062ee866d37246a0b309a85c63ee85"
server_key_version: "encryptionkey@0"
I20260810 18:11:23.556236 24706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:23.575078 24706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:23.577888 24706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:23.578059 24706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:23.584883 24706 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:40023
I20260810 18:11:23.584893 24822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:40023 every 8 connection(s)
I20260810 18:11:23.585786 24706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:23.586457 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24706
I20260810 18:11:23.586566 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:23.586840 23813 external_mini_cluster.cc:1428] Setting key f83bd58867e7a6761c8c2951dfe4af85
I20260810 18:11:23.588932 24824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:23.595252 24824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631: Bootstrap starting.
Aug 10 18:11:23 dist-test-slave-jpj1 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:23.597536 24824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:23.598343 24824 log.cc:824] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:23.600469 24824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631: No bootstrap required, opened a new log
I20260810 18:11:23.600565 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:23.588172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.23.65.126:40023)
0810 18:11:23.588558 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:23.588574 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:23.588610 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:11:23.589268 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:23.589286 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:23.589628 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:23.589903 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:23.590751 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:23.591791 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:23.592891 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:23.593337 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:23.593444 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:23.596166 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:23.596170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:23.596183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:23.596212 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:23.597768 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:23.598207 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:23.598211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:23.598215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:23.598331 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:23.598601 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:23.598604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:23.598605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:23.599091 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:23.599266 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:23.599554 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 18:11:23.599849 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260810 18:11:23.604439 24824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf78dd4f50843ea9165cbe76cf79631" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40023 } }
I20260810 18:11:23.604650 24824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:23.604712 24824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf78dd4f50843ea9165cbe76cf79631, State: Initialized, Role: FOLLOWER
I20260810 18:11:23.605090 24824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf78dd4f50843ea9165cbe76cf79631" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40023 } }
I20260810 18:11:23.605217 24824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:23.605278 24824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:23.605369 24824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:23.606159 24824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf78dd4f50843ea9165cbe76cf79631" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40023 } }
I20260810 18:11:23.606498 24824 leader_election.cc:302] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cf78dd4f50843ea9165cbe76cf79631; no voters:
I20260810 18:11:23.606801 24824 leader_election.cc:288] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:23.606918 24833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:23.607154 24833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [term 1 LEADER]: Becoming Leader. State: Replica: 5cf78dd4f50843ea9165cbe76cf79631, State: Running, Role: LEADER
I20260810 18:11:23.607447 24833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf78dd4f50843ea9165cbe76cf79631" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40023 } }
I20260810 18:11:23.608009 24824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:23.609059 24837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf78dd4f50843ea9165cbe76cf79631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf78dd4f50843ea9165cbe76cf79631" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40023 } } }
I20260810 18:11:23.609059 24839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf78dd4f50843ea9165cbe76cf79631. Latest consensus state: current_term: 1 leader_uuid: "5cf78dd4f50843ea9165cbe76cf79631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf78dd4f50843ea9165cbe76cf79631" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40023 } } }
I20260810 18:11:23.609216 24839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:23.609427 24837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:23.609697 24849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:23.612339 24849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:23.617205 24849 catalog_manager.cc:1347] Generated new cluster ID: 694dd37936e344a69a7bf061b91afb95
I20260810 18:11:23.617270 24849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:23.629865 24849 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:23.630682 24849 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:23.640568 24849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cf78dd4f50843ea9165cbe76cf79631: Generated new TSK 0
I20260810 18:11:23.641321 24849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:11:23.678685 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40023
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:23.792477 24866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:23.792738 24866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:23.792801 24866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:23.796716 24866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:23.796773 24866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:23.796856 24866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:23.802030 24866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40023
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:23.803027 24866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:23.804054 24866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:23.813567 24874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:23.813647 24878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:23.813967 24866 server_base.cc:1034] running on GCE node
W20260810 18:11:23.814031 24875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:23.814459 24866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:23.815030 24866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:23.816172 24866 hybrid_clock.cc:648] HybridClock initialized: now 1786385483816142 us; error 42 us; skew 500 ppm
Aug 10 18:11:23 dist-test-slave-jpj1 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385483, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:23.818754 24866 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:11:23.819600 24866 webserver.cc:460] Webserver started at http://127.23.65.65:34527/ using document root <none> and password file <none>
I20260810 18:11:23.820118 24866 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:23.820185 24866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:23.820366 24866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:23.822302 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "e1bfd48a13f54c599b896d3f48e249f6"
format_stamp: "Formatted at 2026-08-10 18:11:23 on dist-test-slave-jpj1"
server_key: "6f1079e6b5de0d86e775d03fce3033af"
server_key_iv: "ad3710ac8d6611edbcc340daf425fb5c"
server_key_version: "encryptionkey@0"
I20260810 18:11:23.822830 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "e1bfd48a13f54c599b896d3f48e249f6"
format_stamp: "Formatted at 2026-08-10 18:11:23 on dist-test-slave-jpj1"
server_key: "6f1079e6b5de0d86e775d03fce3033af"
server_key_iv: "ad3710ac8d6611edbcc340daf425fb5c"
server_key_version: "encryptionkey@0"
I20260810 18:11:23.826363 24866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:11:23.828745 24891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:23.830080 24866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:23.830183 24866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "e1bfd48a13f54c599b896d3f48e249f6"
format_stamp: "Formatted at 2026-08-10 18:11:23 on dist-test-slave-jpj1"
server_key: "6f1079e6b5de0d86e775d03fce3033af"
server_key_iv: "ad3710ac8d6611edbcc340daf425fb5c"
server_key_version: "encryptionkey@0"
I20260810 18:11:23.830273 24866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:23.842029 24866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:23.844874 24866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:23.845031 24866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:23.845542 24866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:23.846283 24866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:23.846340 24866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:23.846390 24866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:23.846415 24866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:23.854923 24866 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:39137
I20260810 18:11:23.854979 25088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:39137 every 8 connection(s)
I20260810 18:11:23.855855 24866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:23.857842 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24866
I20260810 18:11:23.857930 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:23.858167 23813 external_mini_cluster.cc:1428] Setting key 453a53cc9ff427accd5ffa15e41a1985
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
Aug 10 18:11:23 dist-test-slave-jpj1 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385483, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
I20260810 18:11:23.872319 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:23.857531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:60835 (local address 127.23.65.126:40023)
0810 18:11:23.857689 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:23.857694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:23.858611 (+ 917us) server_negotiation.cc:407] Connection header received
0810 18:11:23.859739 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:23.859742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:23.859795 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:23.859883 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:23.861066 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:23.861738 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:23.863268 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:23.863623 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:23.863758 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:23.868764 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:23.868768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:23.868771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:23.868797 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:23.870090 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:23.870669 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:23.870672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:23.870674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:23.870797 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:23.871174 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:23.871176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:23.871178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:23.871353 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:23.871426 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:23.872041 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 18:11:23.872147 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:11:23.873153 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:23.857927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40023 (local address 127.23.65.65:60835)
0810 18:11:23.858431 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:23.858492 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:23.859497 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:23.860048 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:23.860055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:23.860417 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:23.860931 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:23.860946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:23.861898 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:23.861902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:23.863132 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:23.863144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:23.863793 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:23.863796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:23.863946 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:23.867043 (+ 3097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:23.867060 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:23.868586 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:23.870186 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:23.870191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:23.870202 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:23.870541 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:23.870885 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:23.870888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:23.870891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:23.871085 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:23.871453 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:23.871459 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:23.871796 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 18:11:23.872010 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 18:11:23.872367 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260810 18:11:23.874333 25089 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40023
I20260810 18:11:23.874593 25089 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:23.875180 25089 heartbeater.cc:511] Master 127.23.65.126:40023 requested a full tablet report, sending...
I20260810 18:11:23.876744 24753 ts_manager.cc:194] Registered new tserver with Master: e1bfd48a13f54c599b896d3f48e249f6 (127.23.65.65:39137)
I20260810 18:11:23.877861 24753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:60835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:11:23.886044 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40023
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:23.995554 25099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:23.995805 25099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:23.995847 25099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:23.999775 25099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:23.999830 25099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:23.999886 25099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:24.004985 25099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40023
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:24.005951 25099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:24.006909 25099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:24.016716 25105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:24.016755 25106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:24.016755 25110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:24.016940 25099 server_base.cc:1034] running on GCE node
I20260810 18:11:24.017315 25099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:24.017930 25099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:24.019090 25099 hybrid_clock.cc:648] HybridClock initialized: now 1786385484019073 us; error 33 us; skew 500 ppm
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:24.021780 25099 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:11:24.022612 25099 webserver.cc:460] Webserver started at http://127.23.65.66:44387/ using document root <none> and password file <none>
I20260810 18:11:24.023133 25099 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:24.023197 25099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:24.023394 25099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:24.025401 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "d57a552949f543ecb7259d0c03e4420d"
format_stamp: "Formatted at 2026-08-10 18:11:24 on dist-test-slave-jpj1"
server_key: "20fc5d2be6dc058080bd2f847f5ddb8d"
server_key_iv: "493b0fb0f93788508d09df71de2a8ece"
server_key_version: "encryptionkey@0"
I20260810 18:11:24.025933 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "d57a552949f543ecb7259d0c03e4420d"
format_stamp: "Formatted at 2026-08-10 18:11:24 on dist-test-slave-jpj1"
server_key: "20fc5d2be6dc058080bd2f847f5ddb8d"
server_key_iv: "493b0fb0f93788508d09df71de2a8ece"
server_key_version: "encryptionkey@0"
I20260810 18:11:24.029479 25099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 18:11:24.031771 25120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:24.032948 25099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:11:24.033058 25099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "d57a552949f543ecb7259d0c03e4420d"
format_stamp: "Formatted at 2026-08-10 18:11:24 on dist-test-slave-jpj1"
server_key: "20fc5d2be6dc058080bd2f847f5ddb8d"
server_key_iv: "493b0fb0f93788508d09df71de2a8ece"
server_key_version: "encryptionkey@0"
I20260810 18:11:24.033154 25099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:24.044421 25099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:24.047463 25099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:24.047691 25099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:24.048233 25099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:24.049003 25099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:24.049052 25099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:24.049103 25099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:24.049131 25099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:24.057621 25099 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:45919
I20260810 18:11:24.057662 25336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:45919 every 8 connection(s)
I20260810 18:11:24.058444 25099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:24.062196 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 25099
I20260810 18:11:24.062310 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:24.062565 23813 external_mini_cluster.cc:1428] Setting key 0ad67701ccf62faaaa9705ae5577f1a7
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:24.072129 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.060191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:44605 (local address 127.23.65.126:40023)
0810 18:11:24.060321 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.060326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.061202 (+ 876us) server_negotiation.cc:407] Connection header received
0810 18:11:24.062041 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.062044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.062105 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.062200 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:24.063371 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.064071 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.065618 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.065845 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.065973 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.068447 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:24.068452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:24.068454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:24.068481 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:24.070003 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.070498 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.070502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.070504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.070602 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.070908 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.070912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.070914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.071147 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:24.071257 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.071808 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.071955 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":44}
I20260810 18:11:24.072844 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.060606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40023 (local address 127.23.65.66:44605)
0810 18:11:24.061050 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:24.061090 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:24.061845 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:24.062366 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:24.062377 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:24.062744 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:24.063231 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.063250 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.064206 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.064209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.065461 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.065474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.066029 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.066033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.066235 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.066770 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:24.066786 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:24.068276 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:24.070119 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:24.070124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:24.070141 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:24.070394 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:24.070688 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:24.070690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:24.070691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:24.070821 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:24.071274 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:24.071280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:24.071509 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 18:11:24.071736 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 18:11:24.071995 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:11:24.073963 25337 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40023
I20260810 18:11:24.074223 25337 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:24.074705 25337 heartbeater.cc:511] Master 127.23.65.126:40023 requested a full tablet report, sending...
I20260810 18:11:24.075755 24753 ts_manager.cc:194] Registered new tserver with Master: d57a552949f543ecb7259d0c03e4420d (127.23.65.66:45919)
I20260810 18:11:24.076298 24753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:44605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:11:24.090422 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40023
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:24.204135 25345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:24.204397 25345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:24.204458 25345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:24.208393 25345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:24.208446 25345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:24.208519 25345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:24.214293 25345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40023
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:24.215260 25345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:24.216277 25345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:24.225622 25355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:24.225652 25364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:24.225924 25345 server_base.cc:1034] running on GCE node
W20260810 18:11:24.225941 25359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:24.226346 25345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:24.226922 25345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:24.228071 25345 hybrid_clock.cc:648] HybridClock initialized: now 1786385484228045 us; error 39 us; skew 500 ppm
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:24.230717 25345 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:11:24.231549 25345 webserver.cc:460] Webserver started at http://127.23.65.67:43149/ using document root <none> and password file <none>
I20260810 18:11:24.232097 25345 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:24.232165 25345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:24.232348 25345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:24.234367 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad"
format_stamp: "Formatted at 2026-08-10 18:11:24 on dist-test-slave-jpj1"
server_key: "0b86b971f0b7be5a4240311469ba79bb"
server_key_iv: "a22ce24ccc903c50a357bf0c2f34f3b0"
server_key_version: "encryptionkey@0"
I20260810 18:11:24.234928 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad"
format_stamp: "Formatted at 2026-08-10 18:11:24 on dist-test-slave-jpj1"
server_key: "0b86b971f0b7be5a4240311469ba79bb"
server_key_iv: "a22ce24ccc903c50a357bf0c2f34f3b0"
server_key_version: "encryptionkey@0"
I20260810 18:11:24.238651 25345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:11:24.241060 25373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:24.242214 25345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:24.242319 25345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad"
format_stamp: "Formatted at 2026-08-10 18:11:24 on dist-test-slave-jpj1"
server_key: "0b86b971f0b7be5a4240311469ba79bb"
server_key_iv: "a22ce24ccc903c50a357bf0c2f34f3b0"
server_key_version: "encryptionkey@0"
I20260810 18:11:24.242410 25345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:24.252166 25345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:24.255160 25345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:24.255331 25345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:24.255932 25345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:24.256736 25345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:24.256786 25345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:24.256839 25345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:24.256863 25345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:24.273979 25345 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:38683
I20260810 18:11:24.274091 25561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:38683 every 8 connection(s)
I20260810 18:11:24.274994 25345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:24.276588 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 25345
I20260810 18:11:24.276692 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:24.276949 23813 external_mini_cluster.cc:1428] Setting key 21ac935bda9d9470686a1b3e43905391
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:24.289882 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.277093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:39409 (local address 127.23.65.126:40023)
0810 18:11:24.277255 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.277259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.279217 (+ 1958us) server_negotiation.cc:407] Connection header received
0810 18:11:24.280409 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.280411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.280456 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.280522 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:24.281938 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.282458 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.283858 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.284072 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.284146 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.286726 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:24.286729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:24.286731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:24.286756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:24.288030 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.288467 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.288470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.288471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.288539 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.288786 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.288788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.288789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.288948 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:24.289017 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.289632 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.289755 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:11:24.290931 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.277544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40023 (local address 127.23.65.67:39409)
0810 18:11:24.279092 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:24.279144 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:24.280228 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:24.280655 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:24.280661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:24.281169 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:24.281819 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.281836 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.282577 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.282580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.283764 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.283774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.284190 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.284192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.284373 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.284945 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:24.284962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:24.286561 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:24.288140 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:24.288145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:24.288157 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:24.288396 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:24.288598 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:24.288600 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:24.288601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:24.288724 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:24.289062 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:24.289068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:24.289330 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 18:11:24.289551 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 18:11:24.289885 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1294,"thread_start_us":194,"threads_started":1}
I20260810 18:11:24.291962 25564 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40023
I20260810 18:11:24.292191 25564 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:24.292678 25564 heartbeater.cc:511] Master 127.23.65.126:40023 requested a full tablet report, sending...
I20260810 18:11:24.293608 24753 ts_manager.cc:194] Registered new tserver with Master: d7bad8a52dfc4b82b5b7eeb5cb848fad (127.23.65.67:38683)
I20260810 18:11:24.294137 24753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:39409
I20260810 18:11:24.302402 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:11:24.314105 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.306632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.23.65.126:40023)
0810 18:11:24.306737 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.306740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.307208 (+ 468us) server_negotiation.cc:407] Connection header received
0810 18:11:24.307307 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.307309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.307352 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.307405 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:24.308130 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.308706 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.309623 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.309852 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.309926 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.311309 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:24.311313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:24.311315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:24.311338 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:24.312681 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.313077 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.313080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.313081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.313167 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.313378 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.313380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.313381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.313590 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:24.313714 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.313848 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.313950 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:11:24.319528 24753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:24.323009 24753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:24.340075 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.332911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:39137 (local address 127.0.0.1:48526)
0810 18:11:24.333292 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:24.333341 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:24.333589 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:24.335017 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:24.335020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:24.335035 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:24.335109 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.335114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.336671 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.336674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.338053 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.338065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.339434 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.339438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.339683 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.339709 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:11:24.339776 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 18:11:24.339925 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":92,"thread_start_us":68,"threads_started":1}
I20260810 18:11:24.340088 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.334256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.23.65.67:38683)
0810 18:11:24.334879 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.334925 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.334951 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:11:24.335027 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.335032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.335216 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.335506 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:24.335735 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.337340 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.338670 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.339240 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.339373 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.339611 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.339823 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.339925 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":231,"threads_started":1}
I20260810 18:11:24.340327 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.333591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.23.65.66:45919)
0810 18:11:24.334029 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.334077 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.334107 (+ 30us) server_negotiation.cc:407] Connection header received
0810 18:11:24.334263 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.334269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.334487 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.334883 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:24.335135 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.336595 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.338068 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.339294 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.339393 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.339922 (+ 529us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.339972 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.340051 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":154,"threads_started":1}
I20260810 18:11:24.340734 25590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.333271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:45919 (local address 127.0.0.1:50506)
0810 18:11:24.333949 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:24.333962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:24.334142 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:24.334868 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:24.334874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:24.334901 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:24.335044 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.335051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.336848 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.336851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.337950 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.337960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.339604 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.339607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.339730 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.339748 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:11:24.339799 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 18:11:24.340601 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":29,"thread_start_us":174,"threads_started":1}
I20260810 18:11:24.341012 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.333620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:38683 (local address 127.0.0.1:51982)
0810 18:11:24.333997 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:24.334008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:24.334196 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:24.335488 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:24.335491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:24.335511 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:24.335629 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.335636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.337555 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.337559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.338492 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.338500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.339387 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.339391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.339587 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.339683 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 18:11:24.339781 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 18:11:24.340902 (+ 1121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":1083,"thread_start_us":161,"threads_started":1}
I20260810 18:11:24.341117 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.333059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.23.65.65:39137)
0810 18:11:24.334388 (+ 1329us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.334452 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.334475 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:11:24.334551 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.334556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.334762 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.335230 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:24.335417 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.336443 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.338185 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.339281 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.340575 (+ 1294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.340732 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.340782 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.340869 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228,"thread_start_us":124,"threads_started":1}
I20260810 18:11:24.343956 25457 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6d6f00b4c042ad99a5223812469b0c (DEFAULT_TABLE table=test-table [id=2a1c49393c8a423d8ce7bcb6c7f89ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:24.343959 25203 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6d6f00b4c042ad99a5223812469b0c (DEFAULT_TABLE table=test-table [id=2a1c49393c8a423d8ce7bcb6c7f89ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:24.344019 24970 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6d6f00b4c042ad99a5223812469b0c (DEFAULT_TABLE table=test-table [id=2a1c49393c8a423d8ce7bcb6c7f89ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:24.345196 25457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6d6f00b4c042ad99a5223812469b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:24.347515 25203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6d6f00b4c042ad99a5223812469b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:24.347687 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6d6f00b4c042ad99a5223812469b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:24.352409 25594 tablet_bootstrap.cc:492] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad: Bootstrap starting.
I20260810 18:11:24.355429 25594 tablet_bootstrap.cc:654] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:24.355479 25597 tablet_bootstrap.cc:492] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d: Bootstrap starting.
I20260810 18:11:24.357321 25603 tablet_bootstrap.cc:492] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6: Bootstrap starting.
I20260810 18:11:24.357662 25594 log.cc:824] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:24.358274 25597 tablet_bootstrap.cc:654] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:24.359181 25597 log.cc:824] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:24.359651 25594 tablet_bootstrap.cc:492] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad: No bootstrap required, opened a new log
I20260810 18:11:24.359856 25594 ts_tablet_manager.cc:1397] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260810 18:11:24.360378 25603 tablet_bootstrap.cc:654] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:24.361886 25597 tablet_bootstrap.cc:492] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d: No bootstrap required, opened a new log
I20260810 18:11:24.362141 25597 ts_tablet_manager.cc:1397] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260810 18:11:24.362241 25603 log.cc:824] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:24.364621 25603 tablet_bootstrap.cc:492] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6: No bootstrap required, opened a new log
I20260810 18:11:24.364807 25603 ts_tablet_manager.cc:1397] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:11:24.364907 25594 raft_consensus.cc:348] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.365204 25594 raft_consensus.cc:374] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:24.365267 25594 raft_consensus.cc:729] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7bad8a52dfc4b82b5b7eeb5cb848fad, State: Initialized, Role: FOLLOWER
I20260810 18:11:24.365674 25594 consensus_queue.cc:260] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.366493 25564 heartbeater.cc:503] Master 127.23.65.126:40023 was elected leader, sending a full tablet report...
I20260810 18:11:24.366680 25594 ts_tablet_manager.cc:1428] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
I20260810 18:11:24.366241 25597 raft_consensus.cc:348] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.367044 25597 raft_consensus.cc:374] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:24.367115 25597 raft_consensus.cc:729] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57a552949f543ecb7259d0c03e4420d, State: Initialized, Role: FOLLOWER
I20260810 18:11:24.367493 25597 consensus_queue.cc:260] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.368328 25337 heartbeater.cc:503] Master 127.23.65.126:40023 was elected leader, sending a full tablet report...
I20260810 18:11:24.368497 25597 ts_tablet_manager.cc:1428] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:11:24.371671 25603 raft_consensus.cc:348] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.371929 25603 raft_consensus.cc:374] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:24.371992 25603 raft_consensus.cc:729] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bfd48a13f54c599b896d3f48e249f6, State: Initialized, Role: FOLLOWER
I20260810 18:11:24.372329 25603 consensus_queue.cc:260] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.373034 25089 heartbeater.cc:503] Master 127.23.65.126:40023 was elected leader, sending a full tablet report...
I20260810 18:11:24.373176 25603 ts_tablet_manager.cc:1428] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20260810 18:11:24.413380 25611 raft_consensus.cc:482] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:24.413600 25611 raft_consensus.cc:504] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.414788 25611 leader_election.cc:288] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7bad8a52dfc4b82b5b7eeb5cb848fad (127.23.65.67:38683), d57a552949f543ecb7259d0c03e4420d (127.23.65.66:45919)
I20260810 18:11:24.419384 25610 raft_consensus.cc:482] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:24.419734 25610 raft_consensus.cc:504] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.420053 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.415102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:45631 (local address 127.23.65.67:38683)
0810 18:11:24.415251 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.415256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.415565 (+ 309us) server_negotiation.cc:407] Connection header received
0810 18:11:24.415714 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.415718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.415772 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.415845 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:24.416201 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.416921 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.418651 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.419343 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.419464 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.419519 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.419812 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.419908 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:11:24.420069 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.414983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:38683 (local address 127.23.65.65:45631)
0810 18:11:24.415329 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:24.415345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:24.415604 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:24.415874 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:24.415877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:24.415891 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:24.416088 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.416097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.417035 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.417038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.418528 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.418540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.419473 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.419477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.419651 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.419689 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 18:11:24.419775 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 18:11:24.419852 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":101,"threads_started":1}
I20260810 18:11:24.420137 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.415309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:50783 (local address 127.23.65.66:45919)
0810 18:11:24.415450 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.415455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.415714 (+ 259us) server_negotiation.cc:407] Connection header received
0810 18:11:24.415845 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.415848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.415898 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.415967 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:24.416413 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.417107 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.418792 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.419422 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.419517 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.419587 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.419958 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.420034 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:11:24.420518 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.415037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:45919 (local address 127.23.65.65:50783)
0810 18:11:24.415575 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:24.415589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:24.415748 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:24.415986 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:24.415989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:24.416002 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:24.416302 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.416311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.417210 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.417213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.418673 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.418682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.419614 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.419618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.419803 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.419830 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:11:24.420246 (+ 416us) client_negotiation.cc:240] Negotiation successful
0810 18:11:24.420327 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":139,"threads_started":1}
I20260810 18:11:24.421075 25492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d6f00b4c042ad99a5223812469b0c" candidate_uuid: "e1bfd48a13f54c599b896d3f48e249f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" is_pre_election: true
I20260810 18:11:24.421334 25245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d6f00b4c042ad99a5223812469b0c" candidate_uuid: "e1bfd48a13f54c599b896d3f48e249f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57a552949f543ecb7259d0c03e4420d" is_pre_election: true
I20260810 18:11:24.421528 25245 raft_consensus.cc:2454] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bfd48a13f54c599b896d3f48e249f6 in term 0.
I20260810 18:11:24.421520 25492 raft_consensus.cc:2454] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bfd48a13f54c599b896d3f48e249f6 in term 0.
I20260810 18:11:24.422169 24895 leader_election.cc:302] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57a552949f543ecb7259d0c03e4420d, e1bfd48a13f54c599b896d3f48e249f6; no voters:
I20260810 18:11:24.422441 25611 raft_consensus.cc:2781] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:24.422565 25610 leader_election.cc:288] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bfd48a13f54c599b896d3f48e249f6 (127.23.65.65:39137), d7bad8a52dfc4b82b5b7eeb5cb848fad (127.23.65.67:38683)
I20260810 18:11:24.422593 25611 raft_consensus.cc:482] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:24.422747 25611 raft_consensus.cc:3037] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:24.424834 25611 raft_consensus.cc:504] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.425500 25245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d6f00b4c042ad99a5223812469b0c" candidate_uuid: "e1bfd48a13f54c599b896d3f48e249f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57a552949f543ecb7259d0c03e4420d"
I20260810 18:11:24.425524 25492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d6f00b4c042ad99a5223812469b0c" candidate_uuid: "e1bfd48a13f54c599b896d3f48e249f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad"
I20260810 18:11:24.425619 25245 raft_consensus.cc:3037] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:24.425619 25492 raft_consensus.cc:3037] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:24.426635 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.421963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:39137 (local address 127.23.65.66:48601)
0810 18:11:24.422732 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:24.422747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:24.422836 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:24.423194 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:24.423196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:24.423216 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:24.423483 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.423492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.424398 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.424401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.425277 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.425288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.426181 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.426185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.426353 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.426376 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:11:24.426440 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 18:11:24.426499 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703}
I20260810 18:11:24.427585 25492 raft_consensus.cc:2454] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bfd48a13f54c599b896d3f48e249f6 in term 1.
I20260810 18:11:24.427585 25245 raft_consensus.cc:2454] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bfd48a13f54c599b896d3f48e249f6 in term 1.
I20260810 18:11:24.427937 24892 leader_election.cc:302] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7bad8a52dfc4b82b5b7eeb5cb848fad, e1bfd48a13f54c599b896d3f48e249f6; no voters:
I20260810 18:11:24.428097 25611 leader_election.cc:288] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [CANDIDATE]: Term 1 election: Requested vote from peers d7bad8a52dfc4b82b5b7eeb5cb848fad (127.23.65.67:38683), d57a552949f543ecb7259d0c03e4420d (127.23.65.66:45919)
I20260810 18:11:24.428211 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.421954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:48601 (local address 127.23.65.65:39137)
0810 18:11:24.422055 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.422059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.422919 (+ 860us) server_negotiation.cc:407] Connection header received
0810 18:11:24.422961 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.422963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.423023 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.423081 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:24.423615 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.424306 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.425420 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.425980 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.426084 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.426126 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.428031 (+ 1905us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.428105 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:11:24.428256 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.423289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:57755 (local address 127.23.65.67:38683)
0810 18:11:24.423411 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.423414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.423712 (+ 298us) server_negotiation.cc:407] Connection header received
0810 18:11:24.423840 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.423843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.423890 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.423962 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:24.424243 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.424879 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.426903 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.427431 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.427513 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.427584 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.428045 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.428108 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:11:24.428226 25611 raft_consensus.cc:2781] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:24.428329 25611 raft_consensus.cc:686] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [term 1 LEADER]: Becoming Leader. State: Replica: e1bfd48a13f54c599b896d3f48e249f6, State: Running, Role: LEADER
I20260810 18:11:24.428613 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.422745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:38683 (local address 127.23.65.66:57755)
0810 18:11:24.423470 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:24.423481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:24.423739 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:24.423979 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:24.423982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:24.423998 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:24.424131 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.424137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.425032 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.425035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.426770 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:24.426782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.427706 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.427710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:24.427869 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.427894 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:11:24.428397 (+ 503us) client_negotiation.cc:240] Negotiation successful
0810 18:11:24.428457 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":3,"thread_start_us":300,"threads_started":1}
I20260810 18:11:24.428648 25611 consensus_queue.cc:237] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } }
I20260810 18:11:24.429013 25492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d6f00b4c042ad99a5223812469b0c" candidate_uuid: "d57a552949f543ecb7259d0c03e4420d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" is_pre_election: true
I20260810 18:11:24.429229 25492 raft_consensus.cc:2379] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d57a552949f543ecb7259d0c03e4420d in current term 1: Already voted for candidate e1bfd48a13f54c599b896d3f48e249f6 in this term.
I20260810 18:11:24.429502 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d6f00b4c042ad99a5223812469b0c" candidate_uuid: "d57a552949f543ecb7259d0c03e4420d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bfd48a13f54c599b896d3f48e249f6" is_pre_election: true
I20260810 18:11:24.430096 25121 leader_election.cc:302] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d57a552949f543ecb7259d0c03e4420d; no voters: d7bad8a52dfc4b82b5b7eeb5cb848fad, e1bfd48a13f54c599b896d3f48e249f6
I20260810 18:11:24.430349 25610 raft_consensus.cc:2726] T 7c6d6f00b4c042ad99a5223812469b0c P d57a552949f543ecb7259d0c03e4420d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:11:24.432145 24753 catalog_manager.cc:5495] T 7c6d6f00b4c042ad99a5223812469b0c P e1bfd48a13f54c599b896d3f48e249f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bfd48a13f54c599b896d3f48e249f6 (127.23.65.65). New cstate: current_term: 1 leader_uuid: "e1bfd48a13f54c599b896d3f48e249f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bfd48a13f54c599b896d3f48e249f6" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7bad8a52dfc4b82b5b7eeb5cb848fad" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 38683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d57a552949f543ecb7259d0c03e4420d" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45919 } health_report { overall_health: UNKNOWN } } }
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:24.462877 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.453088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.23.65.65:39137)
0810 18:11:24.453216 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.453220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.453658 (+ 438us) server_negotiation.cc:407] Connection header received
0810 18:11:24.453780 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.453782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.453837 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.453901 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:24.454695 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.455187 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.456182 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.456420 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.456499 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.459404 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:24.459420 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:24.459436 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:24.459460 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:24.461393 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.461795 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.461800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.461801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.461935 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.462131 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.462133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.462134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.462372 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:24.462526 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.462650 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.462741 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:24.492625 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.484850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.23.65.65:39137)
0810 18:11:24.485010 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.485014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.485140 (+ 126us) server_negotiation.cc:407] Connection header received
0810 18:11:24.485275 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.485278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.485321 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.485390 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:24.486129 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.486655 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.487443 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.487708 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.487784 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.490066 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:24.490070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:24.490072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:24.490096 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:24.491183 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.491567 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.491571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.491572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.491645 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.491907 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.491909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.491910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.492056 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:24.492153 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.492372 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.492465 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:24.520380 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:24.512327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.23.65.65:39137)
0810 18:11:24.512438 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:24.512442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:24.512596 (+ 154us) server_negotiation.cc:407] Connection header received
0810 18:11:24.512752 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:24.512755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:24.512801 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:24.512867 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:24.513509 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.514004 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.514873 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:24.515086 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:24.515173 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:24.517551 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:24.517554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:24.517556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:24.517579 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:24.518806 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.519199 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.519202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.519203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.519279 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:24.519528 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:24.519553 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:24.519556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:24.519737 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:24.519974 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:24.520094 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 18:11:24.520219 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 18:11:24.522015 24938 tablet_service.cc:2908] Rejecting scan request for tablet 7c6d6f00b4c042ad99a5223812469b0c: Uninitialized: clean time has not yet been initialized
I20260810 18:11:24.524209 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24866
W20260810 18:11:24.531479 25565 tablet.cc:2369] T 7c6d6f00b4c042ad99a5223812469b0c P d7bad8a52dfc4b82b5b7eeb5cb848fad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:24.538141 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 25099
I20260810 18:11:24.551357 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 25345
I20260810 18:11:24.559393 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24706
2026-08-10T18:11:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2431 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[25635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[25635](info): set up 2 sockets
Aug 10 18:11:24 dist-test-slave-jpj1 krb5kdc[25635](info): commencing operation
krb5kdc: starting...
W20260810 18:11:25.713315 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:25 dist-test-slave-jpj1 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:25.834393 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38389
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:38389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:25.944815 25659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:25.945086 25659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:25.945142 25659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:25.949057 25659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:25.949106 25659 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:11:25.949131 25659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:25.949151 25659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:25.949173 25659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:25.949201 25659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:25.954540 25659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:38389
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:25.955571 25659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:25.956579 25659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:25.964835 25673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:25.964896 25659 server_base.cc:1034] running on GCE node
W20260810 18:11:25.964835 25676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:25.964838 25674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:25.965593 25659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:25.966635 25659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:25.967790 25659 hybrid_clock.cc:648] HybridClock initialized: now 1786385485967771 us; error 31 us; skew 500 ppm
Aug 10 18:11:25 dist-test-slave-jpj1 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:25.970432 25659 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:25.971298 25659 webserver.cc:460] Webserver started at http://127.23.65.126:45961/ using document root <none> and password file <none>
I20260810 18:11:25.971864 25659 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:25.971935 25659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:25.972142 25659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:25.974110 25659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "1ddb13b0ca70482894d21519cb9a672a"
format_stamp: "Formatted at 2026-08-10 18:11:25 on dist-test-slave-jpj1"
server_key: "ea1bbe5a3bedb834f5a0b2b6874bab84"
server_key_iv: "6378eefe65f4c691115c8cc4cecbed66"
server_key_version: "encryptionkey@0"
I20260810 18:11:25.974659 25659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "1ddb13b0ca70482894d21519cb9a672a"
format_stamp: "Formatted at 2026-08-10 18:11:25 on dist-test-slave-jpj1"
server_key: "ea1bbe5a3bedb834f5a0b2b6874bab84"
server_key_iv: "6378eefe65f4c691115c8cc4cecbed66"
server_key_version: "encryptionkey@0"
I20260810 18:11:25.978313 25659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 18:11:25.980602 25690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:25.981812 25659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:25.981935 25659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "1ddb13b0ca70482894d21519cb9a672a"
format_stamp: "Formatted at 2026-08-10 18:11:25 on dist-test-slave-jpj1"
server_key: "ea1bbe5a3bedb834f5a0b2b6874bab84"
server_key_iv: "6378eefe65f4c691115c8cc4cecbed66"
server_key_version: "encryptionkey@0"
I20260810 18:11:25.982029 25659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:26.005183 25659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:26.008106 25659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:26.008292 25659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:26.017715 25659 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:38389
I20260810 18:11:26.017755 25811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:38389 every 8 connection(s)
I20260810 18:11:26.018697 25659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:26.020923 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 25659
I20260810 18:11:26.021044 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:26.021319 23813 external_mini_cluster.cc:1428] Setting key c031947011c7921edf8a989cad6181ae
I20260810 18:11:26.021893 25813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:26.028106 25813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a: Bootstrap starting.
Aug 10 18:11:26 dist-test-slave-jpj1 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:26.030690 25813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:26.031484 25813 log.cc:824] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:26.033640 25813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a: No bootstrap required, opened a new log
I20260810 18:11:26.036427 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.022936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.23.65.126:38389)
0810 18:11:26.023384 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.023400 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.023439 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:11:26.024150 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.024168 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.024501 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.024817 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:26.025659 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.026676 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.027505 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.028035 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.028207 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.030932 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:26.030938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:26.030952 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:26.030982 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:26.033569 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.034064 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.034070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.034074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.034192 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.034469 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.034472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.034473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.034881 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:26.035057 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.035372 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.035693 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
I20260810 18:11:26.036805 25813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb13b0ca70482894d21519cb9a672a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38389 } }
I20260810 18:11:26.037077 25813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:26.037137 25813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ddb13b0ca70482894d21519cb9a672a, State: Initialized, Role: FOLLOWER
I20260810 18:11:26.037580 25813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb13b0ca70482894d21519cb9a672a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38389 } }
I20260810 18:11:26.037719 25813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:26.037786 25813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:26.037882 25813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:26.038847 25813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb13b0ca70482894d21519cb9a672a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38389 } }
I20260810 18:11:26.039160 25813 leader_election.cc:302] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ddb13b0ca70482894d21519cb9a672a; no voters:
I20260810 18:11:26.039445 25813 leader_election.cc:288] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:26.039573 25821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:26.039779 25821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [term 1 LEADER]: Becoming Leader. State: Replica: 1ddb13b0ca70482894d21519cb9a672a, State: Running, Role: LEADER
I20260810 18:11:26.040117 25821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb13b0ca70482894d21519cb9a672a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38389 } }
I20260810 18:11:26.040591 25813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:26.041680 25825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ddb13b0ca70482894d21519cb9a672a. Latest consensus state: current_term: 1 leader_uuid: "1ddb13b0ca70482894d21519cb9a672a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb13b0ca70482894d21519cb9a672a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38389 } } }
I20260810 18:11:26.041818 25825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:26.042088 25822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ddb13b0ca70482894d21519cb9a672a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ddb13b0ca70482894d21519cb9a672a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38389 } } }
I20260810 18:11:26.042269 25822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:26.042177 25835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:26.045374 25835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:26.050881 25835 catalog_manager.cc:1347] Generated new cluster ID: 63f36fe296fc4346a0f6d20103fb9da5
I20260810 18:11:26.050953 25835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:26.075969 25835 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:26.076944 25835 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:26.086585 25835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a: Generated new TSK 0
I20260810 18:11:26.087260 25835 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:11:26.124788 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:38389
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:26.236672 25854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:26.236945 25854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:26.236989 25854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:26.241060 25854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:26.241127 25854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:26.241166 25854 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:11:26.241205 25854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:26.246448 25854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.65.126:38389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:26.247507 25854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:26.248556 25854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:26.258363 25868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:26.258385 25862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:26.258425 25854 server_base.cc:1034] running on GCE node
W20260810 18:11:26.258387 25864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:26.259032 25854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:26.259698 25854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:26.260875 25854 hybrid_clock.cc:648] HybridClock initialized: now 1786385486260847 us; error 45 us; skew 500 ppm
Aug 10 18:11:26 dist-test-slave-jpj1 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:26.263464 25854 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:11:26.264413 25854 webserver.cc:460] Webserver started at http://127.23.65.65:32965/ using document root <none> and password file <none>
I20260810 18:11:26.264962 25854 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:26.265029 25854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:26.265224 25854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:26.267243 25854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "2fc7b980d8af43f9ad5a55628d4c35ac"
format_stamp: "Formatted at 2026-08-10 18:11:26 on dist-test-slave-jpj1"
server_key: "46f8e29a951d156c4f8ff0355ba917bc"
server_key_iv: "00a26cfcbd091e092467514234d6e151"
server_key_version: "encryptionkey@0"
I20260810 18:11:26.267831 25854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "2fc7b980d8af43f9ad5a55628d4c35ac"
format_stamp: "Formatted at 2026-08-10 18:11:26 on dist-test-slave-jpj1"
server_key: "46f8e29a951d156c4f8ff0355ba917bc"
server_key_iv: "00a26cfcbd091e092467514234d6e151"
server_key_version: "encryptionkey@0"
I20260810 18:11:26.271601 25854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:11:26.274045 25878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:26.275233 25854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:11:26.275354 25854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "2fc7b980d8af43f9ad5a55628d4c35ac"
format_stamp: "Formatted at 2026-08-10 18:11:26 on dist-test-slave-jpj1"
server_key: "46f8e29a951d156c4f8ff0355ba917bc"
server_key_iv: "00a26cfcbd091e092467514234d6e151"
server_key_version: "encryptionkey@0"
I20260810 18:11:26.275455 25854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:26.292246 25854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:26.294873 25854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:26.295080 25854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:26.296797 25854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:26.296911 25854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:26.297000 25854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:26.297055 25854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:11:26 dist-test-slave-jpj1 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:26.307518 25854 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:37943
I20260810 18:11:26.307578 26052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:37943 every 8 connection(s)
I20260810 18:11:26.308612 25854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:26.310408 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.297532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:37105 (local address 127.23.65.126:38389)
0810 18:11:26.297687 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.297692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.299028 (+ 1336us) server_negotiation.cc:407] Connection header received
0810 18:11:26.300328 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.300332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.300393 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.300472 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:26.302142 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.302756 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.303962 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.304211 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.304303 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.306787 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:26.306790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:26.306792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:26.306817 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:26.308442 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.308992 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.308996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.308998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.309100 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.309481 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.309484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.309485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.309680 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:26.309762 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.310171 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.310273 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:11:26.311229 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.298349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38389 (local address 127.23.65.65:37105)
0810 18:11:26.298840 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:26.298906 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:26.300045 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:26.300669 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:26.300678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:26.301197 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:26.301969 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.301992 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.302885 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.302888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.303846 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.303856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.304376 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.304381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.304529 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.305060 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:26.305076 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:26.306626 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:26.308554 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.308558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.308569 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.308894 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.309216 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.309219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.309221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.309389 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.309772 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:26.309776 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:26.309989 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 18:11:26.310138 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 18:11:26.310386 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
I20260810 18:11:26.311282 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 25854
I20260810 18:11:26.311381 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:26.311738 23813 external_mini_cluster.cc:1428] Setting key 6cd2c8b0bf373f4665a5da1f71833d96
I20260810 18:11:26.312505 26055 heartbeater.cc:347] Connected to a master server at 127.23.65.126:38389
I20260810 18:11:26.312649 26055 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:26.313199 26055 heartbeater.cc:511] Master 127.23.65.126:38389 requested a full tablet report, sending...
I20260810 18:11:26.316488 25718 ts_manager.cc:194] Registered new tserver with Master: 2fc7b980d8af43f9ad5a55628d4c35ac (127.23.65.65:37943)
I20260810 18:11:26.317858 25718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:37105
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:11:26.323945 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.316631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:55499 (local address 127.23.65.126:38389)
0810 18:11:26.316751 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.316755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.316890 (+ 135us) server_negotiation.cc:407] Connection header received
0810 18:11:26.317002 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.317005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.317038 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.317097 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:26.317569 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.318250 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.319396 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.319738 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.319845 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.320956 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:26.320959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:26.320961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:26.320983 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:26.322369 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.322846 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.322849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.322850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.322925 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.323222 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.323225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.323227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.323393 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:26.323468 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.323751 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.323839 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260810 18:11:26.323951 26069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.316556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38389 (local address 127.23.65.65:55499)
0810 18:11:26.316824 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:26.316838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:26.316930 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:26.317133 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:26.317135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:26.317342 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:26.317459 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.317466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.318384 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.318388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.319262 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.319275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.319863 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.319867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.319963 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.320407 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:26.320422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:26.320830 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:26.322500 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.322503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.322506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.322748 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.323005 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.323009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.323010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.323111 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.323486 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:26.323488 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:26.323600 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 18:11:26.323699 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 18:11:26.323807 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:11:26.341998 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:38389
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:26.452292 26073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:26.452567 26073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:26.452610 26073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:26.456524 26073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:26.456575 26073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:26.456611 26073 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:11:26.456641 26073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:26.461683 26073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.65.126:38389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:26.462636 26073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:26.463609 26073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:26.473039 26083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:26.473014 26086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:26.473135 26082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:26.473171 26073 server_base.cc:1034] running on GCE node
I20260810 18:11:26.473701 26073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:26.474308 26073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:26.475468 26073 hybrid_clock.cc:648] HybridClock initialized: now 1786385486475449 us; error 33 us; skew 500 ppm
Aug 10 18:11:26 dist-test-slave-jpj1 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:26.478152 26073 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:11:26.479000 26073 webserver.cc:460] Webserver started at http://127.23.65.66:37985/ using document root <none> and password file <none>
I20260810 18:11:26.479519 26073 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:26.479626 26073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:26.479852 26073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:26.481876 26073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "dd5c1a6d28df4012a90cd443ab53ffd5"
format_stamp: "Formatted at 2026-08-10 18:11:26 on dist-test-slave-jpj1"
server_key: "5c19b1c50fe2619384c5158307aaee7e"
server_key_iv: "b9a9b68bb44f96fe82d9b17c5de3d60e"
server_key_version: "encryptionkey@0"
I20260810 18:11:26.482410 26073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "dd5c1a6d28df4012a90cd443ab53ffd5"
format_stamp: "Formatted at 2026-08-10 18:11:26 on dist-test-slave-jpj1"
server_key: "5c19b1c50fe2619384c5158307aaee7e"
server_key_iv: "b9a9b68bb44f96fe82d9b17c5de3d60e"
server_key_version: "encryptionkey@0"
I20260810 18:11:26.485999 26073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 18:11:26.488376 26099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:26.489681 26073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:26.489787 26073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "dd5c1a6d28df4012a90cd443ab53ffd5"
format_stamp: "Formatted at 2026-08-10 18:11:26 on dist-test-slave-jpj1"
server_key: "5c19b1c50fe2619384c5158307aaee7e"
server_key_iv: "b9a9b68bb44f96fe82d9b17c5de3d60e"
server_key_version: "encryptionkey@0"
I20260810 18:11:26.489888 26073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:26.499374 26073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:26.502400 26073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:26.502575 26073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:26.504055 26073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:26.504107 26073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:26.504158 26073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:26.504215 26073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:11:26 dist-test-slave-jpj1 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:26.514503 26073 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:35363
I20260810 18:11:26.514557 26303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:35363 every 8 connection(s)
I20260810 18:11:26.515437 26073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:26.517380 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.504680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:49909 (local address 127.23.65.126:38389)
0810 18:11:26.504830 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.504835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.506109 (+ 1274us) server_negotiation.cc:407] Connection header received
0810 18:11:26.507298 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.507302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.507363 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.507450 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:26.508997 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.509629 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.510925 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.511159 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.511245 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.513806 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:26.513810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:26.513813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:26.513840 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:26.515235 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.515843 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.515846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.515847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.515941 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.516357 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.516361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.516363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.516575 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:26.516683 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.517152 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.517254 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:11:26.517889 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 26073
I20260810 18:11:26.517984 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:26.518154 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.505453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38389 (local address 127.23.65.66:49909)
0810 18:11:26.505923 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:26.505986 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:26.507065 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:26.507649 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:26.507660 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:26.508130 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:26.508832 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.508852 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.509757 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.509761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.510808 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.510818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.511296 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.511301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.511470 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.512037 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:26.512053 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:26.513650 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:26.515354 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.515359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.515373 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.515738 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.516056 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.516059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.516061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.516245 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.516707 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:26.516713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:26.516933 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 18:11:26.517098 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 18:11:26.517339 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
I20260810 18:11:26.518234 23813 external_mini_cluster.cc:1428] Setting key 76339bef25c84bb9aeef3fa92d80c454
I20260810 18:11:26.519430 26304 heartbeater.cc:347] Connected to a master server at 127.23.65.126:38389
I20260810 18:11:26.521701 26304 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:26.522166 26304 heartbeater.cc:511] Master 127.23.65.126:38389 requested a full tablet report, sending...
I20260810 18:11:26.523065 25719 ts_manager.cc:194] Registered new tserver with Master: dd5c1a6d28df4012a90cd443ab53ffd5 (127.23.65.66:35363)
I20260810 18:11:26.523732 25719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:49909
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:11:26.528893 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.521494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:50035 (local address 127.23.65.126:38389)
0810 18:11:26.521603 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.521607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.521891 (+ 284us) server_negotiation.cc:407] Connection header received
0810 18:11:26.521984 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.521987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.522038 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.522118 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:26.522585 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.523234 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.524332 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.524631 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.524738 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.525936 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:26.525939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:26.525940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:26.525961 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:26.527284 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.527799 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.527801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.527802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.527874 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.528139 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.528141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.528142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.528335 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:26.528406 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.528696 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.528785 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:11:26.528944 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.521412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38389 (local address 127.23.65.66:50035)
0810 18:11:26.521803 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:26.521817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:26.521913 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:26.522142 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:26.522145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:26.522339 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:26.522466 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.522472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.523371 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.523374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.524192 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.524205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.524756 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.524760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.524882 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.525370 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:26.525385 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:26.525790 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:26.527401 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.527404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.527406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.527702 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.527952 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.527955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.527957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.528053 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.528420 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:26.528422 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:26.528516 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 18:11:26.528644 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 18:11:26.528774 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":2,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:11:26.546504 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:38389
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:26.657511 26316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:26.657745 26316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:26.657786 26316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:26.661612 26316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:26.661660 26316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:26.661696 26316 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:11:26.661731 26316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:26.667035 26316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.65.126:38389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:26.667981 26316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:26.668931 26316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:26.678323 26331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:26.678387 26316 server_base.cc:1034] running on GCE node
W20260810 18:11:26.678448 26325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:26.678311 26323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:26.678985 26316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:26.679643 26316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:26.680802 26316 hybrid_clock.cc:648] HybridClock initialized: now 1786385486680781 us; error 35 us; skew 500 ppm
Aug 10 18:11:26 dist-test-slave-jpj1 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:26.683499 26316 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:11:26.684302 26316 webserver.cc:460] Webserver started at http://127.23.65.67:39461/ using document root <none> and password file <none>
I20260810 18:11:26.684865 26316 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:26.684985 26316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:26.685218 26316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:26.687203 26316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "78716827426246a3a942e8c4c0c30cde"
format_stamp: "Formatted at 2026-08-10 18:11:26 on dist-test-slave-jpj1"
server_key: "06c717e46855316bc083c78407624c39"
server_key_iv: "3a6757c1fec3809afcad247c6863e157"
server_key_version: "encryptionkey@0"
I20260810 18:11:26.687767 26316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "78716827426246a3a942e8c4c0c30cde"
format_stamp: "Formatted at 2026-08-10 18:11:26 on dist-test-slave-jpj1"
server_key: "06c717e46855316bc083c78407624c39"
server_key_iv: "3a6757c1fec3809afcad247c6863e157"
server_key_version: "encryptionkey@0"
I20260810 18:11:26.691731 26316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:11:26.694128 26345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:26.695297 26316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:26.695405 26316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "78716827426246a3a942e8c4c0c30cde"
format_stamp: "Formatted at 2026-08-10 18:11:26 on dist-test-slave-jpj1"
server_key: "06c717e46855316bc083c78407624c39"
server_key_iv: "3a6757c1fec3809afcad247c6863e157"
server_key_version: "encryptionkey@0"
I20260810 18:11:26.695495 26316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:26.719973 26316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:26.723228 26316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:26.723418 26316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:26.725065 26316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:26.725132 26316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:26.725189 26316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:26.725258 26316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 10 18:11:26 dist-test-slave-jpj1 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:26.736241 26316 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:44731
I20260810 18:11:26.736279 26515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:44731 every 8 connection(s)
I20260810 18:11:26.737275 26316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:26.738880 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.726021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:41295 (local address 127.23.65.126:38389)
0810 18:11:26.726220 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.726227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.727932 (+ 1705us) server_negotiation.cc:407] Connection header received
0810 18:11:26.728957 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.728960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.729020 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.729096 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:26.730311 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.730841 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.732240 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.732597 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.732728 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.735201 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:26.735205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:26.735208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:26.735240 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:26.736814 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.737334 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.737338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.737340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.737439 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.737765 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.737769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.737771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.738016 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:26.738117 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.738648 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.738762 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 18:11:26.739673 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.726604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38389 (local address 127.23.65.67:41295)
0810 18:11:26.727741 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:26.727811 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:26.728737 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:26.729271 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:26.729278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:26.729607 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:26.730164 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.730181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.730963 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.730966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.732070 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.732082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.732795 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.732800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.733002 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.733569 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:26.733585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:26.735034 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:26.736925 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.736929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.736939 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.737221 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.737529 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.737532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.737533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.737668 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.738173 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:26.738179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:26.738420 (+ 241us) client_negotiation.cc:769] Sending connection context
0810 18:11:26.738575 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 18:11:26.738810 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":147,"threads_started":1}
I20260810 18:11:26.740816 26516 heartbeater.cc:347] Connected to a master server at 127.23.65.126:38389
I20260810 18:11:26.740964 26516 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:26.741465 26516 heartbeater.cc:511] Master 127.23.65.126:38389 requested a full tablet report, sending...
I20260810 18:11:26.742465 25716 ts_manager.cc:194] Registered new tserver with Master: 78716827426246a3a942e8c4c0c30cde (127.23.65.67:44731)
I20260810 18:11:26.742997 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 26316
I20260810 18:11:26.743113 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:26.743185 25716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:41295
I20260810 18:11:26.743404 23813 external_mini_cluster.cc:1428] Setting key 2ced3dce427f1b41eaa9edae2d486613
I20260810 18:11:26.745335 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:11:26.750166 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.742964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:40479 (local address 127.23.65.126:38389)
0810 18:11:26.743485 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.743490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.743503 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:26.743560 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.743562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.743603 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.743677 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:26.744264 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.745028 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.745939 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.746164 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.746248 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.747211 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:26.747215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:26.747217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:26.747241 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:26.748646 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.749028 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.749031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.749033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.749122 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.749398 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.749401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.749403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.749570 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:26.749668 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.749917 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.750016 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457}
I20260810 18:11:26.750174 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.742916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38389 (local address 127.23.65.67:40479)
0810 18:11:26.743238 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:26.743252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:26.743346 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:26.743725 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:26.743907 (+ 182us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:26.744066 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:26.744164 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.744171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.745183 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.745186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.745825 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.745835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.746261 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.746264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.746353 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.746771 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:26.746783 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:26.747082 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:26.748751 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.748754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.748755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.748941 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.749228 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:26.749231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:26.749233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:26.749313 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:26.749677 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:26.749680 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:26.749764 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 18:11:26.749849 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:11:26.749983 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
Aug 10 18:11:26 dist-test-slave-jpj1 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385486, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:11:26 dist-test-slave-jpj1 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385486, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:26.771500 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.764204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.23.65.126:38389)
0810 18:11:26.764341 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.764344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.764356 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:11:26.764390 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.764393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.764429 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.764496 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:26.765200 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.765714 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.766521 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.766733 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.766821 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.769060 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:26.769064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:26.769066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:26.769090 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:26.770231 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.770579 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.770581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.770582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.770647 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:26.770833 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:26.770835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:26.770836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:26.770991 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:26.771081 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.771247 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.771345 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:11:26.774330 25716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:26.777356 25716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:26.790876 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.785954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.23.65.65:37943)
0810 18:11:26.786220 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.786235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.786764 (+ 529us) server_negotiation.cc:407] Connection header received
0810 18:11:26.786929 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.786935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.787114 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.787257 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:26.787488 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.788374 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.789544 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.790214 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.790321 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.790493 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.790641 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.790731 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260810 18:11:26.790866 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.785298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37943 (local address 127.0.0.1:49132)
0810 18:11:26.786562 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:26.786575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:26.786655 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:26.787248 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:26.787251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:26.787269 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:26.787382 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.787387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.788480 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.788484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.789409 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.789420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.790331 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.790335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.790457 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.790499 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 18:11:26.790600 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 18:11:26.790720 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1206,"mutex_wait_us":46,"thread_start_us":169,"threads_started":1}
I20260810 18:11:26.791064 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.785177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:35363 (local address 127.0.0.1:44398)
0810 18:11:26.785631 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:26.785681 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:26.785801 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:26.786342 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:26.786349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:26.786371 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:26.786508 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.786516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.787968 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.787971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.789398 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.789409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.790661 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.790665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.790775 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.790798 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:11:26.790864 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:11:26.790930 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":21,"thread_start_us":70,"threads_started":1}
I20260810 18:11:26.791448 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.785341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44398 (local address 127.23.65.66:35363)
0810 18:11:26.785809 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.785832 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.785957 (+ 125us) server_negotiation.cc:407] Connection header received
0810 18:11:26.786027 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.786031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.786183 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.786323 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:26.786648 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.787851 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.789532 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.790361 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.790450 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.790599 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.791212 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.791307 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":128,"threads_started":1}
I20260810 18:11:26.791991 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.786325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:44731 (local address 127.0.0.1:36568)
0810 18:11:26.787024 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:26.787037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:26.787121 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:26.787609 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:26.787612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:26.787630 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:26.787727 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.787734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.789045 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.789049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.790230 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:26.790241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.791629 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.791633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:26.791720 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.791737 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 18:11:26.791803 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:11:26.791870 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":43,"thread_start_us":151,"threads_started":1}
I20260810 18:11:26.792073 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:26.786489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.23.65.67:44731)
0810 18:11:26.786897 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:26.786908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:26.787218 (+ 310us) server_negotiation.cc:407] Connection header received
0810 18:11:26.787295 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:26.787300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:26.787473 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:26.787634 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:26.787812 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.788929 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.790372 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:26.791311 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:26.791405 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:26.791566 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:26.791850 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 18:11:26.791937 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":109,"threads_started":1}
I20260810 18:11:26.794265 26418 tablet_service.cc:1467] Processing CreateTablet for tablet fa1d1942ea074a118de93c66715914d6 (DEFAULT_TABLE table=test-table [id=ff3d4d65d15c4d89abbfd00e18a746fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:26.794176 26193 tablet_service.cc:1467] Processing CreateTablet for tablet fa1d1942ea074a118de93c66715914d6 (DEFAULT_TABLE table=test-table [id=ff3d4d65d15c4d89abbfd00e18a746fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:26.794169 25962 tablet_service.cc:1467] Processing CreateTablet for tablet fa1d1942ea074a118de93c66715914d6 (DEFAULT_TABLE table=test-table [id=ff3d4d65d15c4d89abbfd00e18a746fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:26.795202 26418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1d1942ea074a118de93c66715914d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:26.795239 25962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1d1942ea074a118de93c66715914d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:26.795239 26193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1d1942ea074a118de93c66715914d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:26.801916 26565 tablet_bootstrap.cc:492] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: Bootstrap starting.
I20260810 18:11:26.803627 26567 tablet_bootstrap.cc:492] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: Bootstrap starting.
I20260810 18:11:26.804210 26568 tablet_bootstrap.cc:492] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: Bootstrap starting.
I20260810 18:11:26.804651 26565 tablet_bootstrap.cc:654] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:26.805366 26565 log.cc:824] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:26.805739 26567 tablet_bootstrap.cc:654] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:26.806247 26568 tablet_bootstrap.cc:654] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:26.806631 26567 log.cc:824] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:26.807052 26568 log.cc:824] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:26.807211 26565 tablet_bootstrap.cc:492] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: No bootstrap required, opened a new log
I20260810 18:11:26.807415 26565 ts_tablet_manager.cc:1397] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:11:26.808671 26567 tablet_bootstrap.cc:492] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: No bootstrap required, opened a new log
I20260810 18:11:26.808864 26567 ts_tablet_manager.cc:1397] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:11:26.809312 26568 tablet_bootstrap.cc:492] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: No bootstrap required, opened a new log
I20260810 18:11:26.809491 26568 ts_tablet_manager.cc:1397] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:11:26.810935 26565 raft_consensus.cc:348] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:26.811146 26565 raft_consensus.cc:374] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:26.811216 26565 raft_consensus.cc:729] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78716827426246a3a942e8c4c0c30cde, State: Initialized, Role: FOLLOWER
I20260810 18:11:26.811650 26565 consensus_queue.cc:260] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:26.812570 26565 ts_tablet_manager.cc:1428] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:11:26.812747 26567 raft_consensus.cc:348] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:26.812915 26516 heartbeater.cc:503] Master 127.23.65.126:38389 was elected leader, sending a full tablet report...
I20260810 18:11:26.812987 26567 raft_consensus.cc:374] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:26.813042 26567 raft_consensus.cc:729] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc7b980d8af43f9ad5a55628d4c35ac, State: Initialized, Role: FOLLOWER
I20260810 18:11:26.813496 26567 consensus_queue.cc:260] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:26.814486 26567 ts_tablet_manager.cc:1428] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:11:26.814577 26055 heartbeater.cc:503] Master 127.23.65.126:38389 was elected leader, sending a full tablet report...
I20260810 18:11:26.814734 26568 raft_consensus.cc:348] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:26.814929 26568 raft_consensus.cc:374] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:26.814960 26568 raft_consensus.cc:729] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5c1a6d28df4012a90cd443ab53ffd5, State: Initialized, Role: FOLLOWER
I20260810 18:11:26.815271 26568 consensus_queue.cc:260] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:26.815984 26304 heartbeater.cc:503] Master 127.23.65.126:38389 was elected leader, sending a full tablet report...
I20260810 18:11:26.816298 26568 ts_tablet_manager.cc:1428] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
W20260810 18:11:26.988479 26517 tablet.cc:2369] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:27.003082 26580 raft_consensus.cc:482] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:27.003306 26580 raft_consensus.cc:504] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.004547 26580 leader_election.cc:288] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fc7b980d8af43f9ad5a55628d4c35ac (127.23.65.65:37943), 78716827426246a3a942e8c4c0c30cde (127.23.65.67:44731)
I20260810 18:11:27.008628 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.004926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:38525 (local address 127.23.65.65:37943)
0810 18:11:27.005073 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.005078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.005285 (+ 207us) server_negotiation.cc:407] Connection header received
0810 18:11:27.005413 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.005415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.005462 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.005543 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:27.005805 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.006317 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.007372 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.008033 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.008150 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.008209 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.008451 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.008521 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:11:27.008723 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.004890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37943 (local address 127.23.65.66:38525)
0810 18:11:27.005178 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:27.005190 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:27.005306 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:27.005565 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:27.005568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:27.005585 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:27.005693 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.005698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.006421 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.006424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.007233 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.007243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.008149 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.008152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.008301 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.008344 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 18:11:27.008426 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:11:27.008503 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":33,"thread_start_us":95,"threads_started":1}
I20260810 18:11:27.009527 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.004926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:55079 (local address 127.23.65.67:44731)
0810 18:11:27.005169 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.005173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.005188 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:27.005348 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.005352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.005404 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.005463 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:27.005748 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.006353 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.008389 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.009017 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.009133 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.009183 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.009363 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.009434 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260810 18:11:27.009557 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.004851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:44731 (local address 127.23.65.66:55079)
0810 18:11:27.005072 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:27.005088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:27.005208 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:27.005488 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:27.005491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:27.005508 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:27.005636 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.005643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.006470 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.006473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.008267 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.008277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.009133 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.009136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.009233 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.009256 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:11:27.009346 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:11:27.009421 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":24}
I20260810 18:11:27.009646 25992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1d1942ea074a118de93c66715914d6" candidate_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" is_pre_election: true
I20260810 18:11:27.010039 25992 raft_consensus.cc:2454] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd5c1a6d28df4012a90cd443ab53ffd5 in term 0.
I20260810 18:11:27.010155 26451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1d1942ea074a118de93c66715914d6" candidate_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78716827426246a3a942e8c4c0c30cde" is_pre_election: true
I20260810 18:11:27.010457 26451 raft_consensus.cc:2454] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd5c1a6d28df4012a90cd443ab53ffd5 in term 0.
I20260810 18:11:27.010617 26109 leader_election.cc:302] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc7b980d8af43f9ad5a55628d4c35ac, dd5c1a6d28df4012a90cd443ab53ffd5; no voters:
I20260810 18:11:27.010960 26580 raft_consensus.cc:2781] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:27.011059 26580 raft_consensus.cc:482] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:27.011109 26580 raft_consensus.cc:3037] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:27.012205 26580 raft_consensus.cc:504] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.012584 26580 leader_election.cc:288] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [CANDIDATE]: Term 1 election: Requested vote from peers 2fc7b980d8af43f9ad5a55628d4c35ac (127.23.65.65:37943), 78716827426246a3a942e8c4c0c30cde (127.23.65.67:44731)
I20260810 18:11:27.012923 25992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1d1942ea074a118de93c66715914d6" candidate_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac"
I20260810 18:11:27.012923 26451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1d1942ea074a118de93c66715914d6" candidate_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78716827426246a3a942e8c4c0c30cde"
I20260810 18:11:27.013052 26451 raft_consensus.cc:3037] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:27.013052 25992 raft_consensus.cc:3037] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:27.014865 25992 raft_consensus.cc:2454] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd5c1a6d28df4012a90cd443ab53ffd5 in term 1.
I20260810 18:11:27.015203 26109 leader_election.cc:302] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc7b980d8af43f9ad5a55628d4c35ac, dd5c1a6d28df4012a90cd443ab53ffd5; no voters:
I20260810 18:11:27.015367 26580 raft_consensus.cc:2781] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:27.015581 26451 raft_consensus.cc:2454] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd5c1a6d28df4012a90cd443ab53ffd5 in term 1.
I20260810 18:11:27.015604 26580 raft_consensus.cc:686] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 LEADER]: Becoming Leader. State: Replica: dd5c1a6d28df4012a90cd443ab53ffd5, State: Running, Role: LEADER
I20260810 18:11:27.015949 26580 consensus_queue.cc:237] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
W20260810 18:11:27.016767 26306 tablet.cc:2369] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:27.018982 25716 catalog_manager.cc:5495] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd5c1a6d28df4012a90cd443ab53ffd5 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } health_report { overall_health: UNKNOWN } } }
I20260810 18:11:27.037752 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.034139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.23.65.66:35363)
0810 18:11:27.034263 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.034268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.034282 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:27.034494 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.034496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.034549 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.034629 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:27.034892 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.035403 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.036312 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.036525 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.036611 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.036942 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:27.037368 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:27.037470 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.037566 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.037623 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:11:27.044130 25992 raft_consensus.cc:1261] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Refusing update from remote peer dd5c1a6d28df4012a90cd443ab53ffd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:27.044879 26582 consensus_queue.cc:1035] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:27.046155 26451 raft_consensus.cc:1261] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Refusing update from remote peer dd5c1a6d28df4012a90cd443ab53ffd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:27.046757 26582 consensus_queue.cc:1035] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:27.052066 26586 mvcc.cc:205] Tried to move back new op lower bound from 7317034954924552192 to 7317034954820239360. Current Snapshot: MvccSnapshot[applied={T|T < 7317034954924552192}]
I20260810 18:11:27.053452 26587 mvcc.cc:205] Tried to move back new op lower bound from 7317034954924552192 to 7317034954820239360. Current Snapshot: MvccSnapshot[applied={T|T < 7317034954924552192}]
I20260810 18:11:27.056655 26589 mvcc.cc:205] Tried to move back new op lower bound from 7317034954924552192 to 7317034954820239360. Current Snapshot: MvccSnapshot[applied={T|T < 7317034954924552192}]
I20260810 18:11:27.066627 25719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:27.067584 25719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:27.070341 25719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a: Sending DeleteTablet for 3 replicas of tablet fa1d1942ea074a118de93c66715914d6
I20260810 18:11:27.071201 26418 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1d1942ea074a118de93c66715914d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:27 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:36568
I20260810 18:11:27.071642 26193 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1d1942ea074a118de93c66715914d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:27 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:44398
I20260810 18:11:27.072085 26601 tablet_replica.cc:330] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: stopping tablet replica
I20260810 18:11:27.072229 25962 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1d1942ea074a118de93c66715914d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:27 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:49132
I20260810 18:11:27.072255 25719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:11:27.072619 26601 raft_consensus.cc:2229] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 18:11:27.072786 25719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:27.072865 26604 tablet_replica.cc:330] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: stopping tablet replica
I20260810 18:11:27.073146 26601 raft_consensus.cc:2258] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:27.073158 26604 raft_consensus.cc:2229] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:27.073539 26604 raft_consensus.cc:2258] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:27.073899 26603 tablet_replica.cc:330] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: stopping tablet replica
I20260810 18:11:27.074193 26603 raft_consensus.cc:2229] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:27.074565 26603 raft_consensus.cc:2258] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:27.075594 26604 ts_tablet_manager.cc:1905] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:27.075814 26601 ts_tablet_manager.cc:1905] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:27.078449 26193 tablet_service.cc:1467] Processing CreateTablet for tablet 43de64e8ae33436681b0af2faa0ea5fd (DEFAULT_TABLE table=test-table [id=c66c0712d4254e75ba3bd00c463a8ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:27.078783 26193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43de64e8ae33436681b0af2faa0ea5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:27.079262 26601 ts_tablet_manager.cc:1918] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:27.079378 26601 log.cc:1196] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/fa1d1942ea074a118de93c66715914d6
I20260810 18:11:27.079727 26601 ts_tablet_manager.cc:1939] T fa1d1942ea074a118de93c66715914d6 P 78716827426246a3a942e8c4c0c30cde: Deleting consensus metadata
I20260810 18:11:27.079849 26604 ts_tablet_manager.cc:1918] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:27.079999 26604 log.cc:1196] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/fa1d1942ea074a118de93c66715914d6
I20260810 18:11:27.080082 26603 ts_tablet_manager.cc:1905] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:27.080410 26604 ts_tablet_manager.cc:1939] T fa1d1942ea074a118de93c66715914d6 P 2fc7b980d8af43f9ad5a55628d4c35ac: Deleting consensus metadata
I20260810 18:11:27.081655 25962 tablet_service.cc:1467] Processing CreateTablet for tablet 43de64e8ae33436681b0af2faa0ea5fd (DEFAULT_TABLE table=test-table [id=c66c0712d4254e75ba3bd00c463a8ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:27.081734 25697 catalog_manager.cc:4841] TS 2fc7b980d8af43f9ad5a55628d4c35ac (127.23.65.65:37943): tablet fa1d1942ea074a118de93c66715914d6 (table test-table [id=ff3d4d65d15c4d89abbfd00e18a746fa]) successfully deleted
I20260810 18:11:27.081929 25962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43de64e8ae33436681b0af2faa0ea5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:27.082248 26568 tablet_bootstrap.cc:492] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: Bootstrap starting.
I20260810 18:11:27.083843 26603 ts_tablet_manager.cc:1918] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:27.083947 26603 log.cc:1196] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/fa1d1942ea074a118de93c66715914d6
I20260810 18:11:27.084208 26568 tablet_bootstrap.cc:654] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:27.084270 26603 ts_tablet_manager.cc:1939] T fa1d1942ea074a118de93c66715914d6 P dd5c1a6d28df4012a90cd443ab53ffd5: Deleting consensus metadata
I20260810 18:11:27.084450 26567 tablet_bootstrap.cc:492] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: Bootstrap starting.
I20260810 18:11:27.085521 25695 catalog_manager.cc:4841] TS 78716827426246a3a942e8c4c0c30cde (127.23.65.67:44731): tablet fa1d1942ea074a118de93c66715914d6 (table test-table [id=ff3d4d65d15c4d89abbfd00e18a746fa]) successfully deleted
I20260810 18:11:27.085669 25692 catalog_manager.cc:4841] TS dd5c1a6d28df4012a90cd443ab53ffd5 (127.23.65.66:35363): tablet fa1d1942ea074a118de93c66715914d6 (table test-table [id=ff3d4d65d15c4d89abbfd00e18a746fa]) successfully deleted
I20260810 18:11:27.086098 26418 tablet_service.cc:1467] Processing CreateTablet for tablet 43de64e8ae33436681b0af2faa0ea5fd (DEFAULT_TABLE table=test-table [id=c66c0712d4254e75ba3bd00c463a8ed8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:27.086270 26567 tablet_bootstrap.cc:654] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:27.086375 26418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43de64e8ae33436681b0af2faa0ea5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:27.087070 26568 tablet_bootstrap.cc:492] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: No bootstrap required, opened a new log
I20260810 18:11:27.087170 26568 ts_tablet_manager.cc:1397] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 18:11:27.087920 26568 raft_consensus.cc:348] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.088048 26568 raft_consensus.cc:374] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:27.088088 26568 raft_consensus.cc:729] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5c1a6d28df4012a90cd443ab53ffd5, State: Initialized, Role: FOLLOWER
I20260810 18:11:27.088204 26568 consensus_queue.cc:260] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.088472 26568 ts_tablet_manager.cc:1428] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 18:11:27.089021 26565 tablet_bootstrap.cc:492] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: Bootstrap starting.
I20260810 18:11:27.089118 26567 tablet_bootstrap.cc:492] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: No bootstrap required, opened a new log
I20260810 18:11:27.089217 26567 ts_tablet_manager.cc:1397] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260810 18:11:27.090072 26567 raft_consensus.cc:348] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.090195 26567 raft_consensus.cc:374] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:27.090232 26567 raft_consensus.cc:729] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc7b980d8af43f9ad5a55628d4c35ac, State: Initialized, Role: FOLLOWER
I20260810 18:11:27.090373 26567 consensus_queue.cc:260] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.090639 26567 ts_tablet_manager.cc:1428] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 18:11:27.091015 26565 tablet_bootstrap.cc:654] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:27.093286 26565 tablet_bootstrap.cc:492] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: No bootstrap required, opened a new log
I20260810 18:11:27.093390 26565 ts_tablet_manager.cc:1397] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260810 18:11:27.094156 26565 raft_consensus.cc:348] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.094256 26565 raft_consensus.cc:374] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:27.094287 26565 raft_consensus.cc:729] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78716827426246a3a942e8c4c0c30cde, State: Initialized, Role: FOLLOWER
I20260810 18:11:27.094412 26565 consensus_queue.cc:260] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.094683 26565 ts_tablet_manager.cc:1428] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:27.275434 26582 raft_consensus.cc:482] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:27.275612 26582 raft_consensus.cc:504] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.276018 26582 leader_election.cc:288] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fc7b980d8af43f9ad5a55628d4c35ac (127.23.65.65:37943), 78716827426246a3a942e8c4c0c30cde (127.23.65.67:44731)
I20260810 18:11:27.276556 25992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43de64e8ae33436681b0af2faa0ea5fd" candidate_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" is_pre_election: true
I20260810 18:11:27.276556 26451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43de64e8ae33436681b0af2faa0ea5fd" candidate_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78716827426246a3a942e8c4c0c30cde" is_pre_election: true
I20260810 18:11:27.276728 26451 raft_consensus.cc:2454] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd5c1a6d28df4012a90cd443ab53ffd5 in term 0.
I20260810 18:11:27.276759 25992 raft_consensus.cc:2454] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd5c1a6d28df4012a90cd443ab53ffd5 in term 0.
I20260810 18:11:27.277066 26107 leader_election.cc:302] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78716827426246a3a942e8c4c0c30cde, dd5c1a6d28df4012a90cd443ab53ffd5; no voters:
I20260810 18:11:27.277266 26582 raft_consensus.cc:2781] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:27.277329 26582 raft_consensus.cc:482] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:27.277355 26582 raft_consensus.cc:3037] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:27.278396 26582 raft_consensus.cc:504] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.278769 26582 leader_election.cc:288] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [CANDIDATE]: Term 1 election: Requested vote from peers 2fc7b980d8af43f9ad5a55628d4c35ac (127.23.65.65:37943), 78716827426246a3a942e8c4c0c30cde (127.23.65.67:44731)
I20260810 18:11:27.279160 25992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43de64e8ae33436681b0af2faa0ea5fd" candidate_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac"
I20260810 18:11:27.279171 26451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43de64e8ae33436681b0af2faa0ea5fd" candidate_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78716827426246a3a942e8c4c0c30cde"
I20260810 18:11:27.279299 25992 raft_consensus.cc:3037] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:27.279314 26451 raft_consensus.cc:3037] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:27.280584 26451 raft_consensus.cc:2454] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd5c1a6d28df4012a90cd443ab53ffd5 in term 1.
I20260810 18:11:27.280727 25992 raft_consensus.cc:2454] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd5c1a6d28df4012a90cd443ab53ffd5 in term 1.
I20260810 18:11:27.280967 26107 leader_election.cc:302] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78716827426246a3a942e8c4c0c30cde, dd5c1a6d28df4012a90cd443ab53ffd5; no voters:
I20260810 18:11:27.281152 26582 raft_consensus.cc:2781] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:27.281248 26582 raft_consensus.cc:686] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 LEADER]: Becoming Leader. State: Replica: dd5c1a6d28df4012a90cd443ab53ffd5, State: Running, Role: LEADER
I20260810 18:11:27.281392 26582 consensus_queue.cc:237] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.282994 25719 catalog_manager.cc:5495] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd5c1a6d28df4012a90cd443ab53ffd5 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } health_report { overall_health: UNKNOWN } } }
W20260810 18:11:27.310585 26057 tablet.cc:2369] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 18:11:27 dist-test-slave-jpj1 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:27.361980 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.353472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.23.65.126:38389)
0810 18:11:27.353761 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.353771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.353909 (+ 138us) server_negotiation.cc:407] Connection header received
0810 18:11:27.354042 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.354045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.354084 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.354166 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:27.354922 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.355457 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.356233 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.356444 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.356527 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.358938 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:27.358941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:27.358943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:27.358963 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:27.360409 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:27.360820 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:27.360823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:27.360825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:27.360900 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:27.361133 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:27.361136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:27.361138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:27.361304 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:27.361386 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.361669 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.361804 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260810 18:11:27.361980 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.353390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38389 (local address 127.0.0.1:56410)
0810 18:11:27.353819 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:27.353832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:27.353951 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:27.354195 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:27.354198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:27.354699 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:27.354803 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.354810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.355579 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.355582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.356128 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.356136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.356538 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.356541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.356631 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.357197 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:27.357210 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:27.358813 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:27.360504 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:27.360513 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:27.360520 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:27.360735 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:27.360976 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:27.360979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:27.360981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:27.361053 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:27.361399 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:27.361406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:27.361511 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 18:11:27.361602 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 18:11:27.361755 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":2,"thread_start_us":120,"threads_started":1}
I20260810 18:11:27.366168 25719 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:56410:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:11:27.366935 25719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:27.371677 26193 tablet_service.cc:1467] Processing CreateTablet for tablet a69d9d3ab5064bcea7a86353be35725f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92b583e02e75411fa3e8f6d058b17d8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:11:27.371846 26418 tablet_service.cc:1467] Processing CreateTablet for tablet a69d9d3ab5064bcea7a86353be35725f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92b583e02e75411fa3e8f6d058b17d8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:11:27.372133 26193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69d9d3ab5064bcea7a86353be35725f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:27.371622 25962 tablet_service.cc:1467] Processing CreateTablet for tablet a69d9d3ab5064bcea7a86353be35725f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92b583e02e75411fa3e8f6d058b17d8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:11:27.372306 26418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69d9d3ab5064bcea7a86353be35725f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:27.372575 25962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69d9d3ab5064bcea7a86353be35725f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:27.375069 26568 tablet_bootstrap.cc:492] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: Bootstrap starting.
I20260810 18:11:27.375788 26565 tablet_bootstrap.cc:492] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: Bootstrap starting.
I20260810 18:11:27.375788 26567 tablet_bootstrap.cc:492] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: Bootstrap starting.
I20260810 18:11:27.376266 26568 tablet_bootstrap.cc:654] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:27.377120 26567 tablet_bootstrap.cc:654] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:27.377223 26565 tablet_bootstrap.cc:654] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:27.377565 26568 tablet_bootstrap.cc:492] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: No bootstrap required, opened a new log
I20260810 18:11:27.377657 26568 ts_tablet_manager.cc:1397] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 18:11:27.378087 26568 raft_consensus.cc:348] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.378214 26568 raft_consensus.cc:374] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:27.378253 26568 raft_consensus.cc:729] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5c1a6d28df4012a90cd443ab53ffd5, State: Initialized, Role: FOLLOWER
I20260810 18:11:27.378402 26568 consensus_queue.cc:260] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.378571 26567 tablet_bootstrap.cc:492] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: No bootstrap required, opened a new log
I20260810 18:11:27.378621 26565 tablet_bootstrap.cc:492] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: No bootstrap required, opened a new log
I20260810 18:11:27.378664 26567 ts_tablet_manager.cc:1397] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260810 18:11:27.378698 26565 ts_tablet_manager.cc:1397] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260810 18:11:27.378834 26582 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.378957 26582 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:27.379201 26565 raft_consensus.cc:348] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.379226 26567 raft_consensus.cc:348] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.379320 26567 raft_consensus.cc:374] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:27.379320 26565 raft_consensus.cc:374] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:27.379351 26565 raft_consensus.cc:729] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78716827426246a3a942e8c4c0c30cde, State: Initialized, Role: FOLLOWER
I20260810 18:11:27.379351 26567 raft_consensus.cc:729] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc7b980d8af43f9ad5a55628d4c35ac, State: Initialized, Role: FOLLOWER
I20260810 18:11:27.379468 26567 consensus_queue.cc:260] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.379765 26567 ts_tablet_manager.cc:1428] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:27.379839 26579 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.379971 26579 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:27.380200 26568 ts_tablet_manager.cc:1428] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:27.380507 26565 consensus_queue.cc:260] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.380798 26565 ts_tablet_manager.cc:1428] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:27.381404 26578 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.381590 26578 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:27.516280 26578 raft_consensus.cc:482] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:27.516424 26578 raft_consensus.cc:504] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.517536 26578 leader_election.cc:288] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fc7b980d8af43f9ad5a55628d4c35ac (127.23.65.65:37943), dd5c1a6d28df4012a90cd443ab53ffd5 (127.23.65.66:35363)
I20260810 18:11:27.522022 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.517789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37943 (local address 127.23.65.67:43793)
0810 18:11:27.518176 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:27.518190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:27.518293 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:27.518632 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:27.518634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:27.518649 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:27.518749 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.518756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.519855 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.519858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.520856 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.520867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.521584 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.521587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.521688 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.521715 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:11:27.521778 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 18:11:27.521849 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":42,"thread_start_us":90,"threads_started":1}
I20260810 18:11:27.522038 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.518110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:43793 (local address 127.23.65.65:37943)
0810 18:11:27.518378 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.518383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.518401 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:11:27.518463 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.518466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.518515 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.518595 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:27.518861 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.519735 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.520980 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.521472 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.521602 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.521663 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.521807 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.521886 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260810 18:11:27.522511 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.517944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:42123 (local address 127.23.65.66:35363)
0810 18:11:27.518066 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.518071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.518484 (+ 413us) server_negotiation.cc:407] Connection header received
0810 18:11:27.518590 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.518592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.518640 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.518711 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:27.519094 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.519759 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.521283 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.521836 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.521945 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.521992 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.522322 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.522403 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 18:11:27.522619 25992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69d9d3ab5064bcea7a86353be35725f" candidate_uuid: "78716827426246a3a942e8c4c0c30cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" is_pre_election: true
I20260810 18:11:27.522760 25992 raft_consensus.cc:2454] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78716827426246a3a942e8c4c0c30cde in term 0.
I20260810 18:11:27.522827 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.517793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:35363 (local address 127.23.65.67:42123)
0810 18:11:27.518387 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:27.518400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:27.518502 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:27.518727 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:27.518729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:27.518742 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:27.518971 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.518979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.519893 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.519897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.521158 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.521169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.522057 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.522060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.522176 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.522199 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:11:27.522645 (+ 446us) client_negotiation.cc:240] Negotiation successful
0810 18:11:27.522721 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":263,"thread_start_us":78,"threads_started":1}
I20260810 18:11:27.523319 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69d9d3ab5064bcea7a86353be35725f" candidate_uuid: "78716827426246a3a942e8c4c0c30cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" is_pre_election: true
I20260810 18:11:27.523352 26349 leader_election.cc:302] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc7b980d8af43f9ad5a55628d4c35ac, 78716827426246a3a942e8c4c0c30cde; no voters:
I20260810 18:11:27.523561 26227 raft_consensus.cc:2454] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78716827426246a3a942e8c4c0c30cde in term 0.
I20260810 18:11:27.523623 26578 raft_consensus.cc:2781] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:27.523679 26578 raft_consensus.cc:482] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:27.523717 26578 raft_consensus.cc:3037] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:27.524941 26578 raft_consensus.cc:504] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.525280 26578 leader_election.cc:288] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [CANDIDATE]: Term 1 election: Requested vote from peers 2fc7b980d8af43f9ad5a55628d4c35ac (127.23.65.65:37943), dd5c1a6d28df4012a90cd443ab53ffd5 (127.23.65.66:35363)
I20260810 18:11:27.525597 25992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69d9d3ab5064bcea7a86353be35725f" candidate_uuid: "78716827426246a3a942e8c4c0c30cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac"
I20260810 18:11:27.525600 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69d9d3ab5064bcea7a86353be35725f" candidate_uuid: "78716827426246a3a942e8c4c0c30cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5"
I20260810 18:11:27.525702 26227 raft_consensus.cc:3037] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:27.525710 25992 raft_consensus.cc:3037] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:27.526672 25992 raft_consensus.cc:2454] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78716827426246a3a942e8c4c0c30cde in term 1.
I20260810 18:11:27.526806 26227 raft_consensus.cc:2454] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78716827426246a3a942e8c4c0c30cde in term 1.
I20260810 18:11:27.526978 26349 leader_election.cc:302] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc7b980d8af43f9ad5a55628d4c35ac, 78716827426246a3a942e8c4c0c30cde; no voters:
I20260810 18:11:27.527196 26578 raft_consensus.cc:2781] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:27.527361 26578 raft_consensus.cc:686] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [term 1 LEADER]: Becoming Leader. State: Replica: 78716827426246a3a942e8c4c0c30cde, State: Running, Role: LEADER
I20260810 18:11:27.527629 26578 consensus_queue.cc:237] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } }
I20260810 18:11:27.528847 26632 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: TxnStatusTablet state changed. Reason: New leader 78716827426246a3a942e8c4c0c30cde. Latest consensus state: current_term: 1 leader_uuid: "78716827426246a3a942e8c4c0c30cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.529013 26632 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:11:27.529842 25718 catalog_manager.cc:5495] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde reported cstate change: term changed from 0 to 1, leader changed from <none> to 78716827426246a3a942e8c4c0c30cde (127.23.65.67). New cstate: current_term: 1 leader_uuid: "78716827426246a3a942e8c4c0c30cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } health_report { overall_health: HEALTHY } } }
I20260810 18:11:27.531890 26635 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:11:27.532140 26635 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:11:27.571748 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.567643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:44731 (local address 127.0.0.1:36580)
0810 18:11:27.567793 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:27.567808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:27.567914 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:27.568323 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:27.568326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:27.568333 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:27.568436 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.568442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.569386 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.569389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.569993 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.570001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.570511 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.570515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.570600 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.570644 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:27.571411 (+ 767us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:27.571428 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 18:11:27.571494 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:11:27.571616 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 18:11:27.571825 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.567730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.23.65.67:44731)
0810 18:11:27.567964 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.567968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.567982 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:27.568058 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.568060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.568155 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.568297 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:27.568536 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.569276 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.570111 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.570402 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.570495 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.570883 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:27.571309 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:27.571419 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.571511 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.571599 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260810 18:11:27.574891 25992 raft_consensus.cc:1261] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Refusing update from remote peer 78716827426246a3a942e8c4c0c30cde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:27.575013 26227 raft_consensus.cc:1261] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 FOLLOWER]: Refusing update from remote peer 78716827426246a3a942e8c4c0c30cde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:27.575467 26632 consensus_queue.cc:1035] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:27.575685 26578 consensus_queue.cc:1035] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:27.577188 26579 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: TxnStatusTablet state changed. Reason: New leader 78716827426246a3a942e8c4c0c30cde. Latest consensus state: current_term: 1 leader_uuid: "78716827426246a3a942e8c4c0c30cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.577319 26579 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:27.577448 26582 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: TxnStatusTablet state changed. Reason: New leader 78716827426246a3a942e8c4c0c30cde. Latest consensus state: current_term: 1 leader_uuid: "78716827426246a3a942e8c4c0c30cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.577569 26582 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:27.579649 26578 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78716827426246a3a942e8c4c0c30cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.579784 26578 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:11:27.580137 26582 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78716827426246a3a942e8c4c0c30cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.580262 26582 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P dd5c1a6d28df4012a90cd443ab53ffd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:27.583674 26632 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78716827426246a3a942e8c4c0c30cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.583786 26632 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P 78716827426246a3a942e8c4c0c30cde: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:11:27.584749 26579 tablet_replica.cc:439] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78716827426246a3a942e8c4c0c30cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 } } peers { permanent_uuid: "dd5c1a6d28df4012a90cd443ab53ffd5" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35363 } } peers { permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 } } }
I20260810 18:11:27.584870 26579 tablet_replica.cc:442] T a69d9d3ab5064bcea7a86353be35725f P 2fc7b980d8af43f9ad5a55628d4c35ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:27.595775 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.592418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:44731 (local address 127.23.65.66:58023)
0810 18:11:27.592683 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:27.592698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:27.592792 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:27.593033 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:27.593035 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:27.593043 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:27.593147 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.593154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.593911 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.593914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.594630 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.594641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.595076 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.595079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.595167 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.595212 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:27.595483 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:27.595490 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:11:27.595567 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 18:11:27.595648 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260810 18:11:27.595763 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.592519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:58023 (local address 127.23.65.67:44731)
0810 18:11:27.592728 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.592733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.592748 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:27.592903 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.592906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.592954 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.593015 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:27.593232 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.593787 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.594739 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.594974 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.595070 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.595336 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:27.595409 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:27.595467 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.595578 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.595629 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 18:11:27.602147 26451 raft_consensus.cc:1261] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Refusing update from remote peer dd5c1a6d28df4012a90cd443ab53ffd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:27.602469 25992 raft_consensus.cc:1261] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Refusing update from remote peer dd5c1a6d28df4012a90cd443ab53ffd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:27.602828 26582 consensus_queue.cc:1035] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc7b980d8af43f9ad5a55628d4c35ac" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:27.602991 26596 consensus_queue.cc:1035] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78716827426246a3a942e8c4c0c30cde" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:27.624634 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.621205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:35363 (local address 127.23.65.67:49295)
0810 18:11:27.621498 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:27.621514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:27.621613 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:27.621813 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:27.621816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:27.621823 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:27.621937 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.621944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.622638 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.622641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.623286 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:27.623297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.623914 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.623917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:27.624019 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.624075 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:27.624357 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:27.624364 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:11:27.624421 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 18:11:27.624499 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260810 18:11:27.624619 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.621476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:49295 (local address 127.23.65.66:35363)
0810 18:11:27.621602 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.621606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.621619 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:27.621693 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.621696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.621738 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.621796 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:27.622035 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.622536 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.623433 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.623788 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.623883 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.624206 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:27.624280 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:27.624338 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.624432 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.624482 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:11:27.646273 25718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:27.646425 25718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:27.648301 25718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ddb13b0ca70482894d21519cb9a672a: Sending DeleteTablet for 3 replicas of tablet 43de64e8ae33436681b0af2faa0ea5fd
I20260810 18:11:27.648792 25962 tablet_service.cc:1514] Processing DeleteTablet for tablet 43de64e8ae33436681b0af2faa0ea5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:27 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:49132
I20260810 18:11:27.648947 26418 tablet_service.cc:1514] Processing DeleteTablet for tablet 43de64e8ae33436681b0af2faa0ea5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:27 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:36568
I20260810 18:11:27.649150 26676 tablet_replica.cc:330] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: stopping tablet replica
I20260810 18:11:27.649178 26193 tablet_service.cc:1514] Processing DeleteTablet for tablet 43de64e8ae33436681b0af2faa0ea5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:27 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:44398
I20260810 18:11:27.649263 26676 raft_consensus.cc:2229] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:27.649355 26676 raft_consensus.cc:2258] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:27.649538 26678 tablet_replica.cc:330] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: stopping tablet replica
I20260810 18:11:27.649670 26678 raft_consensus.cc:2229] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:27.649868 26678 raft_consensus.cc:2258] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:27.650385 26678 ts_tablet_manager.cc:1905] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:27.650386 26676 ts_tablet_manager.cc:1905] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:27.650879 26677 tablet_replica.cc:330] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: stopping tablet replica
I20260810 18:11:27.651014 26677 raft_consensus.cc:2229] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:27.651120 26677 raft_consensus.cc:2258] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:27.651793 26677 ts_tablet_manager.cc:1905] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:27.653389 26676 ts_tablet_manager.cc:1918] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:27.653437 26678 ts_tablet_manager.cc:1918] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:27.653506 26678 log.cc:1196] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/43de64e8ae33436681b0af2faa0ea5fd
I20260810 18:11:27.653615 26676 log.cc:1196] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/43de64e8ae33436681b0af2faa0ea5fd
I20260810 18:11:27.653803 26678 ts_tablet_manager.cc:1939] T 43de64e8ae33436681b0af2faa0ea5fd P dd5c1a6d28df4012a90cd443ab53ffd5: Deleting consensus metadata
I20260810 18:11:27.653980 26676 ts_tablet_manager.cc:1939] T 43de64e8ae33436681b0af2faa0ea5fd P 2fc7b980d8af43f9ad5a55628d4c35ac: Deleting consensus metadata
I20260810 18:11:27.654595 25692 catalog_manager.cc:4841] TS dd5c1a6d28df4012a90cd443ab53ffd5 (127.23.65.66:35363): tablet 43de64e8ae33436681b0af2faa0ea5fd (table test-table [id=c66c0712d4254e75ba3bd00c463a8ed8]) successfully deleted
I20260810 18:11:27.654645 25697 catalog_manager.cc:4841] TS 2fc7b980d8af43f9ad5a55628d4c35ac (127.23.65.65:37943): tablet 43de64e8ae33436681b0af2faa0ea5fd (table test-table [id=c66c0712d4254e75ba3bd00c463a8ed8]) successfully deleted
I20260810 18:11:27.654789 26677 ts_tablet_manager.cc:1918] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:27.654872 26677 log.cc:1196] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/43de64e8ae33436681b0af2faa0ea5fd
I20260810 18:11:27.655117 26677 ts_tablet_manager.cc:1939] T 43de64e8ae33436681b0af2faa0ea5fd P 78716827426246a3a942e8c4c0c30cde: Deleting consensus metadata
I20260810 18:11:27.655771 25695 catalog_manager.cc:4841] TS 78716827426246a3a942e8c4c0c30cde (127.23.65.67:44731): tablet 43de64e8ae33436681b0af2faa0ea5fd (table test-table [id=c66c0712d4254e75ba3bd00c463a8ed8]) successfully deleted
Aug 10 18:11:27 dist-test-slave-jpj1 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385486, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:27.664696 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.655182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49148 (local address 127.23.65.65:37943)
0810 18:11:27.655295 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.655299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.655800 (+ 501us) server_negotiation.cc:407] Connection header received
0810 18:11:27.655992 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.655995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.656046 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.656109 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:27.656865 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.657382 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.658213 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.658445 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.658535 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.661112 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:27.661125 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:27.661140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:27.661163 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:27.662866 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:27.663355 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:27.663361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:27.663363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:27.663464 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:27.663743 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:27.663746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:27.663748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:27.664007 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:27.664215 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.664401 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.664518 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 18:11:27.665190 26047 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49148
I20260810 18:11:27.674041 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:27.666854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.23.65.126:38389)
0810 18:11:27.666972 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:27.666976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:27.666989 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:27.667113 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:27.667115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:27.667152 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:27.667213 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:27.667976 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.668490 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.669530 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:27.669749 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:27.669818 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:27.671243 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:27.671246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:27.671248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:27.671271 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:27.672552 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:27.672986 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:27.672989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:27.672990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:27.673059 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:27.673328 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:27.673330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:27.673331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:27.673492 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:27.673586 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:27.673818 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 18:11:27.673915 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 18:11:27.674403 25718 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56424
I20260810 18:11:27.675472 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 25854
I20260810 18:11:27.716995 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 26073
I20260810 18:11:27.731048 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 26316
I20260810 18:11:27.745951 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 25659
2026-08-10T18:11:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3187 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:27 dist-test-slave-jpj1 krb5kdc[26692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:27 dist-test-slave-jpj1 krb5kdc[26692](info): set up 2 sockets
Aug 10 18:11:27 dist-test-slave-jpj1 krb5kdc[26692](info): commencing operation
krb5kdc: starting...
W20260810 18:11:28.906406 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:29.052057 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:35411
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:29.163812 26714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:29.164085 26714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:29.164124 26714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:29.168038 26714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:29.168087 26714 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:11:29.168104 26714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:29.168121 26714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:29.168141 26714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:29.168160 26714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:29.174647 26714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:35411
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:29.176183 26714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:29.177745 26714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:29.187301 26725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:29.187330 26722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:29.187330 26727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:29.187641 26714 server_base.cc:1034] running on GCE node
I20260810 18:11:29.188203 26714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:29.189276 26714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:29.190452 26714 hybrid_clock.cc:648] HybridClock initialized: now 1786385489190418 us; error 44 us; skew 500 ppm
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:29.193188 26714 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:29.194043 26714 webserver.cc:460] Webserver started at http://127.23.65.126:43671/ using document root <none> and password file <none>
I20260810 18:11:29.194576 26714 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:29.194634 26714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:29.194808 26714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:29.197057 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "4de5d536782047a28392eaf84bd34adb"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "cf2c8345903facb177bcf5b6d0956124"
server_key_iv: "b04b5f55d1512daccf67308fa48477a9"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.197589 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "4de5d536782047a28392eaf84bd34adb"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "cf2c8345903facb177bcf5b6d0956124"
server_key_iv: "b04b5f55d1512daccf67308fa48477a9"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.201284 26714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:11:29.203747 26736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:29.204982 26714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:29.205089 26714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "4de5d536782047a28392eaf84bd34adb"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "cf2c8345903facb177bcf5b6d0956124"
server_key_iv: "b04b5f55d1512daccf67308fa48477a9"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.205195 26714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:29.218901 26714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:29.222035 26714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:29.222343 26714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:29.229969 26714 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:35411
I20260810 18:11:29.229983 26840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:35411 every 8 connection(s)
I20260810 18:11:29.230919 26714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:29.233709 26843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:29.239393 26843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb: Bootstrap starting.
I20260810 18:11:29.240236 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 26714
I20260810 18:11:29.240341 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:29.240572 23813 external_mini_cluster.cc:1428] Setting key e506a96fba15869b5d96df9cfabf4b0e
I20260810 18:11:29.242120 26843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:29.242864 26843 log.cc:824] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:29.245095 26843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb: No bootstrap required, opened a new log
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:29.249290 26843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de5d536782047a28392eaf84bd34adb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35411 } }
I20260810 18:11:29.249547 26843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:29.249624 26843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de5d536782047a28392eaf84bd34adb, State: Initialized, Role: FOLLOWER
I20260810 18:11:29.250067 26843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de5d536782047a28392eaf84bd34adb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35411 } }
I20260810 18:11:29.250247 26843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:29.250319 26843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:29.250423 26843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:29.251557 26843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de5d536782047a28392eaf84bd34adb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35411 } }
I20260810 18:11:29.251935 26843 leader_election.cc:302] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de5d536782047a28392eaf84bd34adb; no voters:
I20260810 18:11:29.252271 26843 leader_election.cc:288] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:29.252400 26854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:29.252638 26854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [term 1 LEADER]: Becoming Leader. State: Replica: 4de5d536782047a28392eaf84bd34adb, State: Running, Role: LEADER
I20260810 18:11:29.252950 26854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de5d536782047a28392eaf84bd34adb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35411 } }
I20260810 18:11:29.253563 26843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:29.254109 26854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de5d536782047a28392eaf84bd34adb. Latest consensus state: current_term: 1 leader_uuid: "4de5d536782047a28392eaf84bd34adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de5d536782047a28392eaf84bd34adb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35411 } } }
I20260810 18:11:29.254226 26854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:29.254621 26862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:29.256309 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.241639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.23.65.126:35411)
0810 18:11:29.242096 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.242112 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.242150 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:11:29.242909 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.242929 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.243323 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.243722 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:29.244599 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.245602 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.246747 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.247206 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.247327 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.250032 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:29.250038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:29.250053 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:29.250085 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:29.252027 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.252534 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.252541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.252547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.252763 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.253153 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.253157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.253159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.253664 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:29.253858 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.254738 (+ 880us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.255052 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":146,"threads_started":1}
I20260810 18:11:29.256330 26855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de5d536782047a28392eaf84bd34adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de5d536782047a28392eaf84bd34adb" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35411 } } }
I20260810 18:11:29.256443 26855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:29.258231 26862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:29.263665 26862 catalog_manager.cc:1347] Generated new cluster ID: 8b7332903be24cfa9c991afaf6f0bda0
I20260810 18:11:29.263736 26862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:29.273352 26862 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:29.274277 26862 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:29.280558 26862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb: Generated new TSK 0
I20260810 18:11:29.281212 26862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:11:29.318441 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:35411
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:29.430343 26893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:29.430593 26893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:29.430637 26893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:29.434561 26893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:29.434610 26893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:29.434652 26893 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:11:29.434665 26893 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:11:29.434698 26893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:29.439661 26893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.65.126:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:29.440562 26893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:29.441478 26893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:29.450910 26900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:29.450982 26893 server_base.cc:1034] running on GCE node
W20260810 18:11:29.451021 26903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:29.451071 26905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:29.451630 26893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:29.452235 26893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:29.453397 26893 hybrid_clock.cc:648] HybridClock initialized: now 1786385489453383 us; error 32 us; skew 500 ppm
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:29.456187 26893 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:11:29.456979 26893 webserver.cc:460] Webserver started at http://127.23.65.65:45125/ using document root <none> and password file <none>
I20260810 18:11:29.457492 26893 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:29.457556 26893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:29.457738 26893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:29.459685 26893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "8ce490c82d5d45c3a19723f7c7239eb9"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "e119455f29de9b46dcb59ec41dfb6c6d"
server_key_iv: "5cef887fa235a2bbe979751e00f0cffe"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.460222 26893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "8ce490c82d5d45c3a19723f7c7239eb9"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "e119455f29de9b46dcb59ec41dfb6c6d"
server_key_iv: "5cef887fa235a2bbe979751e00f0cffe"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.463738 26893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:11:29.466095 26914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:29.467307 26893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:29.467432 26893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "8ce490c82d5d45c3a19723f7c7239eb9"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "e119455f29de9b46dcb59ec41dfb6c6d"
server_key_iv: "5cef887fa235a2bbe979751e00f0cffe"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.467552 26893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:29.482834 26893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:29.485900 26893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:29.486079 26893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:29.487479 26893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:29.487553 26893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:29.487628 26893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:29.487656 26893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:29.498900 26893 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:38293
I20260810 18:11:29.499274 27109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:38293 every 8 connection(s)
I20260810 18:11:29.499902 26893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:29.501789 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.488315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:55209 (local address 127.23.65.126:35411)
0810 18:11:29.488429 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.488432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.489490 (+ 1058us) server_negotiation.cc:407] Connection header received
0810 18:11:29.490497 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.490501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.490562 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.490670 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:29.492172 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.493017 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.494221 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.494526 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.494620 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.498123 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:29.498128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:29.498131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:29.498160 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:29.499768 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.500343 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.500347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.500349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.500447 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.500849 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.500852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.500854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.501038 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:29.501122 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.501556 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.501668 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:11:29.502557 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.488865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:35411 (local address 127.23.65.65:55209)
0810 18:11:29.489345 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:29.489387 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:29.490248 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:29.491092 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:29.491101 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:29.491463 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:29.492010 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.492026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.493156 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.493159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.494065 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.494076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.494692 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.494696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.494855 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.496259 (+ 1404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:29.496280 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:29.497951 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:29.499892 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.499898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.499911 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.500230 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.500563 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.500566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.500568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.500733 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.501126 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:29.501130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:29.501357 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 18:11:29.501509 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 18:11:29.501747 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260810 18:11:29.503860 27110 heartbeater.cc:347] Connected to a master server at 127.23.65.126:35411
I20260810 18:11:29.503999 27110 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:29.504458 27110 heartbeater.cc:511] Master 127.23.65.126:35411 requested a full tablet report, sending...
I20260810 18:11:29.504601 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 26893
I20260810 18:11:29.504724 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:29.505033 23813 external_mini_cluster.cc:1428] Setting key cb336f7503f4b16cf69fb4ee37d14647
I20260810 18:11:29.505981 26761 ts_manager.cc:194] Registered new tserver with Master: 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293)
I20260810 18:11:29.507421 26761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:55209
I20260810 18:11:29.514770 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.505862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:35411 (local address 127.23.65.65:35081)
0810 18:11:29.506129 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:29.506141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:29.506218 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:29.506405 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:29.506407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:29.506559 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:29.506649 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.506653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.507608 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.507612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.508401 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.508411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.510687 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.510690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.510779 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.511216 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:29.511226 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:29.511556 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:29.513267 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.513269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.513271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.513471 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.513855 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.513858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.513859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.513939 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.514279 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:29.514282 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:29.514360 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 18:11:29.514444 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 18:11:29.514577 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":119,"threads_started":1}
I20260810 18:11:29.514765 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.505950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:35081 (local address 127.23.65.126:35411)
0810 18:11:29.506071 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.506075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.506199 (+ 124us) server_negotiation.cc:407] Connection header received
0810 18:11:29.506275 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.506278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.506316 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.506389 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:29.506747 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.507474 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.510352 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.510564 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.510668 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.511688 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:29.511692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:29.511694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:29.511720 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:29.513133 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.513582 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.513586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.513588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.513749 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.514025 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.514029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.514030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.514190 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:29.514275 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.514509 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.514636 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:11:29.534407 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:35411
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:29.644822 27124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:29.645068 27124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:29.645112 27124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:29.648993 27124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:29.649041 27124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:29.649078 27124 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:11:29.649091 27124 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:11:29.649121 27124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:29.654248 27124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.65.126:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:29.655222 27124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:29.656225 27124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:11:29.665535 27124 server_base.cc:1034] running on GCE node
W20260810 18:11:29.665575 27134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:29.665515 27138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:29.665515 27141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:29.666194 27124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:29.666800 27124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:29.667963 27124 hybrid_clock.cc:648] HybridClock initialized: now 1786385489667942 us; error 31 us; skew 500 ppm
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:29.670616 27124 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:11:29.671411 27124 webserver.cc:460] Webserver started at http://127.23.65.66:35715/ using document root <none> and password file <none>
I20260810 18:11:29.671979 27124 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:29.672048 27124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:29.672225 27124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:29.674144 27124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "bb96efa1156e4bb8af60d22470cc6c2a"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "5c036b7ae7155d26554d59e7b5ea818f"
server_key_iv: "0bf3e7001d139931300af35a257e015e"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.674657 27124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "bb96efa1156e4bb8af60d22470cc6c2a"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "5c036b7ae7155d26554d59e7b5ea818f"
server_key_iv: "0bf3e7001d139931300af35a257e015e"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.678212 27124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 18:11:29.680517 27157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:29.681711 27124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:29.681806 27124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "bb96efa1156e4bb8af60d22470cc6c2a"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "5c036b7ae7155d26554d59e7b5ea818f"
server_key_iv: "0bf3e7001d139931300af35a257e015e"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.681890 27124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:29.707290 27124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:29.710243 27124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:29.710397 27124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:29.711819 27124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:29.711870 27124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:29.711913 27124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:29.711941 27124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:29.721844 27124 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:43013
I20260810 18:11:29.722194 27351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:43013 every 8 connection(s)
I20260810 18:11:29.722820 27124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:29.725147 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.712587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:35793 (local address 127.23.65.126:35411)
0810 18:11:29.712754 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.712759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.713596 (+ 837us) server_negotiation.cc:407] Connection header received
0810 18:11:29.714690 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.714693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.714751 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.714833 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:29.716676 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.717263 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.718884 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.719160 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.719257 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.721839 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:29.721843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:29.721846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:29.721875 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:29.723300 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.723857 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.723860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.723861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.723949 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.724246 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.724249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.724250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.724421 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:29.724502 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.724925 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.725028 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:11:29.725893 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.712969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:35411 (local address 127.23.65.66:35793)
0810 18:11:29.713434 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:29.713487 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:29.714434 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:29.715003 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:29.715013 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:29.715401 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:29.716522 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.716541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.717387 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.717391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.718737 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.718751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.719316 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.719320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.719483 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.720112 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:29.720143 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:29.721661 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:29.723426 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.723431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.723445 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.723766 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.724025 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.724027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.724028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.724169 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.724493 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:29.724497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:29.724710 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 18:11:29.724853 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 18:11:29.725083 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
I20260810 18:11:29.726976 27352 heartbeater.cc:347] Connected to a master server at 127.23.65.126:35411
I20260810 18:11:29.727121 27352 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:29.727648 27352 heartbeater.cc:511] Master 127.23.65.126:35411 requested a full tablet report, sending...
I20260810 18:11:29.728565 26761 ts_manager.cc:194] Registered new tserver with Master: bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013)
I20260810 18:11:29.729164 26761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:35793
I20260810 18:11:29.730422 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 27124
I20260810 18:11:29.730535 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:29.730826 23813 external_mini_cluster.cc:1428] Setting key 76294150cd3f770c7f6773cd9fc0aba5
I20260810 18:11:29.736335 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.728959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:35411 (local address 127.23.65.66:47799)
0810 18:11:29.729201 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:29.729215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:29.729292 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:29.729637 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:29.729639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:29.729842 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:29.729953 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.729961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.730861 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.730864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.731615 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.731625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.732064 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.732067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.732164 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.732569 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:29.732583 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:29.732858 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:29.734441 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.734444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.734446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.734678 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.735036 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.735039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.735041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.735160 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.735793 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:29.735796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:29.735915 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 18:11:29.736021 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 18:11:29.736162 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260810 18:11:29.736382 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.729041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:47799 (local address 127.23.65.126:35411)
0810 18:11:29.729436 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.729440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.729451 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:11:29.729482 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.729485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.729526 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.729626 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:29.730053 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.730716 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.731743 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.731968 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.732051 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.732992 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:29.732996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:29.732998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:29.733024 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:29.734341 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.734832 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.734836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.734837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.734932 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.735312 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.735323 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.735324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.735683 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:29.735784 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.736134 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.736244 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:11:29.759428 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:35411
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:29.870119 27370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:29.870352 27370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:29.870396 27370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:29.874339 27370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:29.874389 27370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:29.874426 27370 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:11:29.874444 27370 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:11:29.874475 27370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:29.879433 27370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.65.126:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:29.880385 27370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:29.881302 27370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:29.890784 27381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:29.890779 27378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:29.891042 27370 server_base.cc:1034] running on GCE node
W20260810 18:11:29.890779 27379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:29.891750 27370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:29.892335 27370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:29.893538 27370 hybrid_clock.cc:648] HybridClock initialized: now 1786385489893520 us; error 29 us; skew 500 ppm
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:29.896238 27370 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:11:29.897053 27370 webserver.cc:460] Webserver started at http://127.23.65.67:41667/ using document root <none> and password file <none>
I20260810 18:11:29.897554 27370 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:29.897609 27370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:29.897774 27370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:29.899811 27370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "364c7f10b5434ba691fb306d52b5cfae"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "45d5ad511a8e30a88a95fae34a58ec9a"
server_key_iv: "126c8f83350e554be2fbd0084e4cb2dd"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.900311 27370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "364c7f10b5434ba691fb306d52b5cfae"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "45d5ad511a8e30a88a95fae34a58ec9a"
server_key_iv: "126c8f83350e554be2fbd0084e4cb2dd"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.903908 27370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:29.906169 27395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:29.907402 27370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:29.907505 27370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "364c7f10b5434ba691fb306d52b5cfae"
format_stamp: "Formatted at 2026-08-10 18:11:29 on dist-test-slave-jpj1"
server_key: "45d5ad511a8e30a88a95fae34a58ec9a"
server_key_iv: "126c8f83350e554be2fbd0084e4cb2dd"
server_key_version: "encryptionkey@0"
I20260810 18:11:29.907626 27370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:29.917165 27370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:29.920094 27370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:29.920235 27370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:29.921522 27370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:29.921573 27370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:29.921626 27370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:29.921655 27370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:29.932001 27370 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:37865
I20260810 18:11:29.932045 27598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:37865 every 8 connection(s)
I20260810 18:11:29.933059 27370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:29.935047 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.922271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:34237 (local address 127.23.65.126:35411)
0810 18:11:29.922415 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.922420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.923256 (+ 836us) server_negotiation.cc:407] Connection header received
0810 18:11:29.924345 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.924349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.924408 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.924489 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:29.925920 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.926720 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.928303 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.928582 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.928678 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.931205 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:29.931209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:29.931211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:29.931237 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:29.932971 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.933509 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.933512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.933514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.933606 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.933969 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.933973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.933975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.934160 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:29.934247 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.934737 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.934881 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:11:29.935573 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 27370
I20260810 18:11:29.935698 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:29.935892 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.922642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:35411 (local address 127.23.65.67:34237)
0810 18:11:29.923083 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:29.923131 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:29.924082 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:29.924677 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:29.924686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:29.925113 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:29.925737 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.925758 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.926874 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.926877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.928178 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.928189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.928732 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.928736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.928909 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.929448 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:29.929463 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:29.931034 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:29.933080 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.933084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.933095 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.933381 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.933707 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.933710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.933712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.933876 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.934262 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:29.934269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:29.934506 (+ 237us) client_negotiation.cc:769] Sending connection context
0810 18:11:29.934672 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 18:11:29.934986 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260810 18:11:29.935969 23813 external_mini_cluster.cc:1428] Setting key 6fff877b30a41a82a0bfd0c96072c6b0
I20260810 18:11:29.937254 27599 heartbeater.cc:347] Connected to a master server at 127.23.65.126:35411
I20260810 18:11:29.937405 27599 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:29.937947 27599 heartbeater.cc:511] Master 127.23.65.126:35411 requested a full tablet report, sending...
I20260810 18:11:29.938948 26760 ts_manager.cc:194] Registered new tserver with Master: 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865)
I20260810 18:11:29.939716 26760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:34237
I20260810 18:11:29.945340 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.939190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:48425 (local address 127.23.65.126:35411)
0810 18:11:29.939294 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.939298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.939489 (+ 191us) server_negotiation.cc:407] Connection header received
0810 18:11:29.939606 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.939609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.939651 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.939731 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:29.940179 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.940685 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.941705 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.941910 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.941975 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.942879 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:29.942882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:29.942884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:29.942905 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:29.944016 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.944410 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.944412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.944413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.944476 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.944748 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.944750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.944751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.944903 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:29.944970 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.945153 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.945236 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:11:29.945351 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.939168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:35411 (local address 127.23.65.67:48425)
0810 18:11:29.939414 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:29.939426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:29.939506 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:29.939753 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:29.939755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:29.939947 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:29.940068 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.940077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.940816 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.940819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.941615 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.941625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.941982 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.941985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.942046 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.942433 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:29.942443 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:29.942760 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:29.944144 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.944146 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.944148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.944335 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.944567 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:29.944570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:29.944571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:29.944656 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:29.944967 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:29.944969 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:29.945039 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 18:11:29.945114 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:11:29.945214 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260810 18:11:29.949592 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:11:29 dist-test-slave-jpj1 krb5kdc[26692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:29.975456 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.967454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.23.65.126:35411)
0810 18:11:29.967647 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.967652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.967667 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:29.967709 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.967712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.967755 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.967827 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:29.968753 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.969279 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.970124 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.970348 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.970434 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.972775 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:29.972778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:29.972780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:29.972805 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:29.973934 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.974313 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.974316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.974319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.974399 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:29.974649 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:29.974653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:29.974655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:29.974841 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:29.974963 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.975187 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.975301 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:11:29.978188 26760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:29.981215 26760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:29.995196 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.989522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.23.65.67:37865)
0810 18:11:29.990125 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.990138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.990165 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:11:29.990246 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.990250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.990457 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.991134 (+ 677us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:29.991279 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.992255 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.993737 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.994537 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.994642 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.994825 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.994948 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.995043 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":213,"threads_started":1}
I20260810 18:11:29.995200 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.989349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:37865 (local address 127.0.0.1:51080)
0810 18:11:29.989791 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:29.989837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:29.990071 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:29.990794 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:29.990800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:29.990820 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:29.990958 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.990966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.992364 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.992368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.993576 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.993586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.994653 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.994657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.994763 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.994803 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 18:11:29.994917 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 18:11:29.995049 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":37,"thread_start_us":88,"threads_started":1}
I20260810 18:11:29.995747 27630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.989423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:38293 (local address 127.0.0.1:51872)
0810 18:11:29.990713 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:29.990727 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:29.990808 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:29.991500 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:29.991503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:29.991521 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:29.991655 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.991661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.992788 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.992792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.994289 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.994299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.995246 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.995249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.995487 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.995512 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:11:29.995585 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 18:11:29.995650 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"mutex_wait_us":587,"thread_start_us":112,"threads_started":1}
I20260810 18:11:29.996395 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.990295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.23.65.66:43013)
0810 18:11:29.990743 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.990755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.991202 (+ 447us) server_negotiation.cc:407] Connection header received
0810 18:11:29.991294 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.991298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.991445 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.991616 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:29.991995 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.992859 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.994819 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.995685 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.995786 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.995936 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.996148 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.996255 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":112,"threads_started":1}
I20260810 18:11:29.996855 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.990611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.23.65.65:38293)
0810 18:11:29.991069 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:29.991083 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:29.991102 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:11:29.991163 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:29.991168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:29.991362 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:29.991512 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:29.991766 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.992669 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.994412 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.995116 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.996087 (+ 971us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.996394 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:29.996624 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 18:11:29.996720 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":87,"threads_started":1}
I20260810 18:11:29.998358 27491 tablet_service.cc:1467] Processing CreateTablet for tablet f86b88d02a1b4f4ca95c9265fd0aa147 (DEFAULT_TABLE table=test-table [id=270bc59f617348f392cf1bc630249c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:29.999562 27491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f86b88d02a1b4f4ca95c9265fd0aa147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:29.999492 27009 tablet_service.cc:1467] Processing CreateTablet for tablet f86b88d02a1b4f4ca95c9265fd0aa147 (DEFAULT_TABLE table=test-table [id=270bc59f617348f392cf1bc630249c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:30.000562 27009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f86b88d02a1b4f4ca95c9265fd0aa147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:30.003639 27626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:29.990073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:43013 (local address 127.0.0.1:53726)
0810 18:11:29.991110 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:29.991122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:29.991201 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:29.991755 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:29.991758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:29.991777 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:29.991871 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.991877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.992972 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.992975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.994683 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:29.994694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:29.995816 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:29.995820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:29.995918 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:29.995940 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:11:29.996011 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 18:11:29.996071 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":528,"thread_start_us":109,"threads_started":1}
I20260810 18:11:30.006319 27204 tablet_service.cc:1467] Processing CreateTablet for tablet f86b88d02a1b4f4ca95c9265fd0aa147 (DEFAULT_TABLE table=test-table [id=270bc59f617348f392cf1bc630249c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:30.007192 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f86b88d02a1b4f4ca95c9265fd0aa147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:30.007663 27642 tablet_bootstrap.cc:492] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: Bootstrap starting.
I20260810 18:11:30.008898 27635 tablet_bootstrap.cc:492] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: Bootstrap starting.
I20260810 18:11:30.010073 27642 tablet_bootstrap.cc:654] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:30.011312 27635 tablet_bootstrap.cc:654] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:30.011333 27642 log.cc:824] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:30.012128 27635 log.cc:824] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:30.013551 27642 tablet_bootstrap.cc:492] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: No bootstrap required, opened a new log
I20260810 18:11:30.013772 27642 ts_tablet_manager.cc:1397] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:11:30.013803 27635 tablet_bootstrap.cc:492] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: No bootstrap required, opened a new log
I20260810 18:11:30.013979 27635 ts_tablet_manager.cc:1397] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:11:30.016055 27649 tablet_bootstrap.cc:492] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: Bootstrap starting.
I20260810 18:11:30.017040 27635 raft_consensus.cc:348] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } }
I20260810 18:11:30.017292 27635 raft_consensus.cc:374] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:30.017345 27635 raft_consensus.cc:729] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce490c82d5d45c3a19723f7c7239eb9, State: Initialized, Role: FOLLOWER
I20260810 18:11:30.017632 27642 raft_consensus.cc:348] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } }
I20260810 18:11:30.017884 27642 raft_consensus.cc:374] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:30.017834 27635 consensus_queue.cc:260] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } }
I20260810 18:11:30.017935 27642 raft_consensus.cc:729] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364c7f10b5434ba691fb306d52b5cfae, State: Initialized, Role: FOLLOWER
I20260810 18:11:30.018395 27642 consensus_queue.cc:260] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } }
I20260810 18:11:30.018666 27649 tablet_bootstrap.cc:654] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:30.018805 27635 ts_tablet_manager.cc:1428] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:11:30.019161 27110 heartbeater.cc:503] Master 127.23.65.126:35411 was elected leader, sending a full tablet report...
I20260810 18:11:30.019165 27599 heartbeater.cc:503] Master 127.23.65.126:35411 was elected leader, sending a full tablet report...
I20260810 18:11:30.019594 27649 log.cc:824] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:30.019848 27652 raft_consensus.cc:482] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:30.020026 27652 raft_consensus.cc:504] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } }
I20260810 18:11:30.021257 27652 leader_election.cc:288] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865), bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013)
I20260810 18:11:30.021763 27642 ts_tablet_manager.cc:1428] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 18:11:30.023111 27649 tablet_bootstrap.cc:492] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: No bootstrap required, opened a new log
I20260810 18:11:30.023315 27649 ts_tablet_manager.cc:1397] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:11:30.026096 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.021431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:37865 (local address 127.23.65.65:57549)
0810 18:11:30.021702 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:30.021718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:30.021894 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:30.022246 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:30.022249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:30.022269 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:30.022411 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.022419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.023636 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.023640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.024861 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.024873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.025677 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.025680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.025784 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.025809 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:11:30.025872 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 18:11:30.025945 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
I20260810 18:11:30.026079 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.021818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:57549 (local address 127.23.65.67:37865)
0810 18:11:30.021930 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.021935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.021955 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:11:30.022054 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.022058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.022106 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.022182 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:30.022590 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.023349 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.024991 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.025571 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.025671 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.025723 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.025896 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.025981 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:11:30.026643 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f86b88d02a1b4f4ca95c9265fd0aa147" candidate_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364c7f10b5434ba691fb306d52b5cfae" is_pre_election: true
I20260810 18:11:30.026916 27519 raft_consensus.cc:2454] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce490c82d5d45c3a19723f7c7239eb9 in term 0.
I20260810 18:11:30.027407 26920 leader_election.cc:302] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c7f10b5434ba691fb306d52b5cfae, 8ce490c82d5d45c3a19723f7c7239eb9; no voters:
I20260810 18:11:30.027493 27649 raft_consensus.cc:348] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } }
I20260810 18:11:30.027796 27649 raft_consensus.cc:374] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:30.027784 27652 raft_consensus.cc:2781] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:30.027848 27649 raft_consensus.cc:729] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb96efa1156e4bb8af60d22470cc6c2a, State: Initialized, Role: FOLLOWER
I20260810 18:11:30.027864 27652 raft_consensus.cc:482] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:30.027908 27652 raft_consensus.cc:3037] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:30.028259 27649 consensus_queue.cc:260] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } }
I20260810 18:11:30.028887 27352 heartbeater.cc:503] Master 127.23.65.126:35411 was elected leader, sending a full tablet report...
I20260810 18:11:30.029253 27649 ts_tablet_manager.cc:1428] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:11:30.029240 27652 raft_consensus.cc:504] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } }
I20260810 18:11:30.029482 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.023451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:38159 (local address 127.23.65.66:43013)
0810 18:11:30.023601 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.023605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.025347 (+ 1742us) server_negotiation.cc:407] Connection header received
0810 18:11:30.025528 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.025532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.025581 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.025653 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:30.026021 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.026734 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.028133 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.028717 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.028842 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.028893 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.029302 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.029376 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 18:11:30.029822 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.023005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:43013 (local address 127.23.65.65:38159)
0810 18:11:30.025266 (+ 2261us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:30.025280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:30.025406 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:30.025760 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:30.025764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:30.025784 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:30.025899 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.025906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.026874 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.026878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.027966 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.027978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.028954 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.028958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.029053 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.029076 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:11:30.029145 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:11:30.029222 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2194,"thread_start_us":93,"threads_started":1}
I20260810 18:11:30.029892 27652 leader_election.cc:288] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [CANDIDATE]: Term 1 election: Requested vote from peers 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865), bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013)
I20260810 18:11:30.030009 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f86b88d02a1b4f4ca95c9265fd0aa147" candidate_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364c7f10b5434ba691fb306d52b5cfae"
I20260810 18:11:30.030113 27519 raft_consensus.cc:3037] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:30.030534 27274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f86b88d02a1b4f4ca95c9265fd0aa147" candidate_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" is_pre_election: true
I20260810 18:11:30.030535 27272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f86b88d02a1b4f4ca95c9265fd0aa147" candidate_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb96efa1156e4bb8af60d22470cc6c2a"
I20260810 18:11:30.030792 27274 raft_consensus.cc:2454] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce490c82d5d45c3a19723f7c7239eb9 in term 0.
I20260810 18:11:30.031376 27519 raft_consensus.cc:2454] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce490c82d5d45c3a19723f7c7239eb9 in term 1.
I20260810 18:11:30.031665 26920 leader_election.cc:302] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c7f10b5434ba691fb306d52b5cfae, 8ce490c82d5d45c3a19723f7c7239eb9; no voters:
I20260810 18:11:30.031845 27652 raft_consensus.cc:2781] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:30.031929 27652 raft_consensus.cc:686] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 LEADER]: Becoming Leader. State: Replica: 8ce490c82d5d45c3a19723f7c7239eb9, State: Running, Role: LEADER
I20260810 18:11:30.032181 27652 consensus_queue.cc:237] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } }
I20260810 18:11:30.035105 26760 catalog_manager.cc:5495] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65). New cstate: current_term: 1 leader_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } health_report { overall_health: UNKNOWN } } }
Aug 10 18:11:30 dist-test-slave-jpj1 krb5kdc[26692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:30.069046 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.061178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:35411 (local address 127.0.0.1:48360)
0810 18:11:30.061434 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:30.061445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:30.061554 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:30.061777 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:30.061780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:30.062193 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:30.062300 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.062306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.063019 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.063022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.063601 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.063609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.064010 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.064013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.064087 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.064587 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:30.064599 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:30.065992 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:30.067573 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:30.067583 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:30.067590 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:30.067810 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:30.068051 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:30.068054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:30.068056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:30.068134 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:30.068480 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:30.068488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:30.068595 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 18:11:30.068689 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 18:11:30.068849 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
I20260810 18:11:30.069046 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.061227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.23.65.126:35411)
0810 18:11:30.061370 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.061374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.061515 (+ 141us) server_negotiation.cc:407] Connection header received
0810 18:11:30.061636 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.061639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.061680 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.061769 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:30.062411 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.062923 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.063704 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.063917 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.064000 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.066110 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:30.066113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:30.066115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:30.066135 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:30.067453 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:30.067896 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:30.067900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:30.067902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:30.067973 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:30.068208 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:30.068211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:30.068212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:30.068381 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:30.068464 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.068753 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.068901 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":61}
I20260810 18:11:30.072938 26760 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:48360:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:11:30.073642 26760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:30.078590 27204 tablet_service.cc:1467] Processing CreateTablet for tablet a0831fbd46a44030b36af7a9f0be7ce2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e9601fd526d4eabbbc541b6a44affce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:11:30.078727 27009 tablet_service.cc:1467] Processing CreateTablet for tablet a0831fbd46a44030b36af7a9f0be7ce2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e9601fd526d4eabbbc541b6a44affce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:11:30.079420 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0831fbd46a44030b36af7a9f0be7ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:30.079432 27009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0831fbd46a44030b36af7a9f0be7ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:30.079272 27491 tablet_service.cc:1467] Processing CreateTablet for tablet a0831fbd46a44030b36af7a9f0be7ce2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e9601fd526d4eabbbc541b6a44affce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:11:30.079946 27491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0831fbd46a44030b36af7a9f0be7ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:30.082278 27635 tablet_bootstrap.cc:492] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: Bootstrap starting.
I20260810 18:11:30.082683 27642 tablet_bootstrap.cc:492] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: Bootstrap starting.
I20260810 18:11:30.082691 27649 tablet_bootstrap.cc:492] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: Bootstrap starting.
I20260810 18:11:30.083278 27635 tablet_bootstrap.cc:654] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:30.083693 27642 tablet_bootstrap.cc:654] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:30.083868 27649 tablet_bootstrap.cc:654] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:30.084954 27642 tablet_bootstrap.cc:492] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: No bootstrap required, opened a new log
I20260810 18:11:30.084973 27635 tablet_bootstrap.cc:492] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: No bootstrap required, opened a new log
I20260810 18:11:30.085039 27642 ts_tablet_manager.cc:1397] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 18:11:30.085039 27635 ts_tablet_manager.cc:1397] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:11:30.085218 27649 tablet_bootstrap.cc:492] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: No bootstrap required, opened a new log
I20260810 18:11:30.085275 27649 ts_tablet_manager.cc:1397] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 18:11:30.085469 27635 raft_consensus.cc:348] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.085469 27642 raft_consensus.cc:348] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.085573 27642 raft_consensus.cc:374] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:30.085572 27635 raft_consensus.cc:374] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:30.085604 27642 raft_consensus.cc:729] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364c7f10b5434ba691fb306d52b5cfae, State: Initialized, Role: FOLLOWER
I20260810 18:11:30.085604 27635 raft_consensus.cc:729] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce490c82d5d45c3a19723f7c7239eb9, State: Initialized, Role: FOLLOWER
I20260810 18:11:30.085619 27649 raft_consensus.cc:348] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.085744 27649 raft_consensus.cc:374] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:30.085722 27635 consensus_queue.cc:260] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.085722 27642 consensus_queue.cc:260] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.085780 27649 raft_consensus.cc:729] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb96efa1156e4bb8af60d22470cc6c2a, State: Initialized, Role: FOLLOWER
I20260810 18:11:30.085880 27649 consensus_queue.cc:260] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.085975 27642 ts_tablet_manager.cc:1428] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:11:30.085975 27635 ts_tablet_manager.cc:1428] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:11:30.086140 27649 ts_tablet_manager.cc:1428] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:11:30.086294 27656 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.086424 27656 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:30.086489 27664 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.086645 27664 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:30.085985 27652 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.088474 27652 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 18:11:30.184108 27600 tablet.cc:2369] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:11:30.224246 27353 tablet.cc:2369] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:11:30.251318 27111 tablet.cc:2369] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:30.337041 27664 raft_consensus.cc:482] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:30.337198 27664 raft_consensus.cc:504] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.338438 27664 leader_election.cc:288] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293), 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865)
I20260810 18:11:30.342490 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.338779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:60403 (local address 127.23.65.65:38293)
0810 18:11:30.338912 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.338916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.339136 (+ 220us) server_negotiation.cc:407] Connection header received
0810 18:11:30.339271 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.339274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.339337 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.339416 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:30.339734 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.340336 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.341484 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.341909 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.342017 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.342075 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.342297 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.342373 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:11:30.342491 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.338841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:36201 (local address 127.23.65.67:37865)
0810 18:11:30.338963 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.338967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.339347 (+ 380us) server_negotiation.cc:407] Connection header received
0810 18:11:30.339480 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.339482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.339553 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.339628 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:30.339869 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.340341 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.341498 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.341914 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.342024 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.342076 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.342288 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.342362 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:11:30.342511 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.338695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:38293 (local address 127.23.65.66:60403)
0810 18:11:30.339042 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:30.339054 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:30.339155 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:30.339432 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:30.339435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:30.339455 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:30.339617 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.339626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.340457 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.340461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.341354 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.341364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.342024 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.342028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.342170 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.342196 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:11:30.342258 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:11:30.342333 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":29,"thread_start_us":76,"threads_started":1}
I20260810 18:11:30.342516 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.338715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:37865 (local address 127.23.65.66:36201)
0810 18:11:30.339127 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:30.339143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:30.339381 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:30.339640 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:30.339642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:30.339657 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:30.339767 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.339773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.340469 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.340472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.341386 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.341399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.342029 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.342033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.342167 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.342187 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:11:30.342252 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:11:30.342334 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":76,"thread_start_us":85,"threads_started":1}
I20260810 18:11:30.343252 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0831fbd46a44030b36af7a9f0be7ce2" candidate_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364c7f10b5434ba691fb306d52b5cfae" is_pre_election: true
I20260810 18:11:30.343272 27047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0831fbd46a44030b36af7a9f0be7ce2" candidate_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" is_pre_election: true
I20260810 18:11:30.343393 27519 raft_consensus.cc:2454] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb96efa1156e4bb8af60d22470cc6c2a in term 0.
I20260810 18:11:30.343494 27047 raft_consensus.cc:2454] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb96efa1156e4bb8af60d22470cc6c2a in term 0.
I20260810 18:11:30.343760 27163 leader_election.cc:302] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c7f10b5434ba691fb306d52b5cfae, bb96efa1156e4bb8af60d22470cc6c2a; no voters:
I20260810 18:11:30.344067 27664 raft_consensus.cc:2781] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:30.344139 27664 raft_consensus.cc:482] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:30.344165 27664 raft_consensus.cc:3037] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:30.345153 27664 raft_consensus.cc:504] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.345448 27664 leader_election.cc:288] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [CANDIDATE]: Term 1 election: Requested vote from peers 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293), 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865)
I20260810 18:11:30.345767 27047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0831fbd46a44030b36af7a9f0be7ce2" candidate_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce490c82d5d45c3a19723f7c7239eb9"
I20260810 18:11:30.345794 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0831fbd46a44030b36af7a9f0be7ce2" candidate_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364c7f10b5434ba691fb306d52b5cfae"
I20260810 18:11:30.345886 27047 raft_consensus.cc:3037] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:30.345891 27519 raft_consensus.cc:3037] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:30.346741 27519 raft_consensus.cc:2454] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb96efa1156e4bb8af60d22470cc6c2a in term 1.
I20260810 18:11:30.346850 27047 raft_consensus.cc:2454] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb96efa1156e4bb8af60d22470cc6c2a in term 1.
I20260810 18:11:30.346994 27163 leader_election.cc:302] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c7f10b5434ba691fb306d52b5cfae, bb96efa1156e4bb8af60d22470cc6c2a; no voters:
I20260810 18:11:30.347219 27664 raft_consensus.cc:2781] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:30.347378 27664 raft_consensus.cc:686] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 LEADER]: Becoming Leader. State: Replica: bb96efa1156e4bb8af60d22470cc6c2a, State: Running, Role: LEADER
I20260810 18:11:30.347716 27664 consensus_queue.cc:237] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.349843 27694 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: TxnStatusTablet state changed. Reason: New leader bb96efa1156e4bb8af60d22470cc6c2a. Latest consensus state: current_term: 1 leader_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.349956 27694 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:11:30.350248 27698 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:11:30.350456 27698 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:11:30.351052 26759 catalog_manager.cc:5495] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66). New cstate: current_term: 1 leader_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } health_report { overall_health: UNKNOWN } } }
I20260810 18:11:30.377451 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.373679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:43013 (local address 127.0.0.1:53738)
0810 18:11:30.373915 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:30.373931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:30.374048 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:30.374376 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:30.374379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:30.374387 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:30.374497 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.374502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.375217 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.375221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.375855 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.375865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.376302 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.376305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.376413 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.376459 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.377152 (+ 693us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.377169 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 18:11:30.377234 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:11:30.377329 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260810 18:11:30.377451 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.373806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.23.65.66:43013)
0810 18:11:30.373945 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.373949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.373982 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:11:30.374195 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.374198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.374254 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.374339 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:30.374604 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.375104 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.375979 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.376198 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.376287 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.376597 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.377043 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.377154 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.377245 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.377302 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:11:30.383076 27047 raft_consensus.cc:1261] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Refusing update from remote peer bb96efa1156e4bb8af60d22470cc6c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:30.383100 27519 raft_consensus.cc:1261] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Refusing update from remote peer bb96efa1156e4bb8af60d22470cc6c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:30.383796 27694 consensus_queue.cc:1035] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:30.383982 27664 consensus_queue.cc:1035] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:30.387080 27652 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: TxnStatusTablet state changed. Reason: New leader bb96efa1156e4bb8af60d22470cc6c2a. Latest consensus state: current_term: 1 leader_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.387207 27652 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:30.387432 27656 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: TxnStatusTablet state changed. Reason: New leader bb96efa1156e4bb8af60d22470cc6c2a. Latest consensus state: current_term: 1 leader_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.387583 27656 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:30.391232 27656 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.391278 27652 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.391350 27656 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P 364c7f10b5434ba691fb306d52b5cfae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:30.391376 27652 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P 8ce490c82d5d45c3a19723f7c7239eb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:30.392879 27694 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.393011 27694 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:11:30.392900 27708 tablet_replica.cc:439] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } } }
I20260810 18:11:30.393709 27708 tablet_replica.cc:442] T a0831fbd46a44030b36af7a9f0be7ce2 P bb96efa1156e4bb8af60d22470cc6c2a: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:11:30.403017 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.399205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.23.65.65:38293)
0810 18:11:30.399320 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.399325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.399340 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:30.399448 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.399451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.399508 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.399625 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:30.400001 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.400509 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.401680 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.401910 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.401988 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.402264 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.402658 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.402740 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.402843 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.402890 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:11:30.411237 27652 consensus_queue.cc:1035] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:30.415036 27272 raft_consensus.cc:3037] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:30.416664 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.410653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:45853 (local address 127.23.65.66:43013)
0810 18:11:30.410772 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.410776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.411023 (+ 247us) server_negotiation.cc:407] Connection header received
0810 18:11:30.411199 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.411202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.411256 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.411326 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:30.411616 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.412283 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.413318 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.413623 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.413704 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.414152 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.414239 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.414288 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.416479 (+ 2191us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.416540 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:11:30.416703 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.410558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:43013 (local address 127.23.65.65:45853)
0810 18:11:30.410947 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:30.410962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:30.411092 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:30.411348 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:30.411351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:30.411358 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:30.411493 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.411500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.412395 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.412399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.413177 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.413189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.413817 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.413821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.413915 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.413962 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.415649 (+ 1687us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.415655 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:11:30.416442 (+ 787us) client_negotiation.cc:240] Negotiation successful
0810 18:11:30.416524 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260810 18:11:30.417388 27655 consensus_queue.cc:1035] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:30.443660 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.439907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:56651 (local address 127.23.65.65:38293)
0810 18:11:30.440029 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.440034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.440267 (+ 233us) server_negotiation.cc:407] Connection header received
0810 18:11:30.440380 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.440383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.440431 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.440515 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:30.440774 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.441433 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.442618 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.442879 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.442959 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.443237 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.443302 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.443354 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.443458 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.443502 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:11:30.443725 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.439842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:38293 (local address 127.23.65.66:56651)
0810 18:11:30.440189 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:30.440202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:30.440299 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:30.440554 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:30.440558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:30.440567 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:30.440676 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.440682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.441519 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.441523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.442501 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.442511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.442972 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.442976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.443055 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.443106 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.443373 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.443381 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:11:30.443451 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:11:30.443561 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":97,"threads_started":1}
I20260810 18:11:30.468366 26759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:30.468576 26759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:30.471328 26759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb: Sending DeleteTablet for 3 replicas of tablet f86b88d02a1b4f4ca95c9265fd0aa147
I20260810 18:11:30.472081 27491 tablet_service.cc:1514] Processing DeleteTablet for tablet f86b88d02a1b4f4ca95c9265fd0aa147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:30 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:51080
I20260810 18:11:30.472345 27204 tablet_service.cc:1514] Processing DeleteTablet for tablet f86b88d02a1b4f4ca95c9265fd0aa147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:30 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:53726
I20260810 18:11:30.472841 27009 tablet_service.cc:1514] Processing DeleteTablet for tablet f86b88d02a1b4f4ca95c9265fd0aa147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:30 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:51872
I20260810 18:11:30.473336 27741 tablet_replica.cc:330] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: stopping tablet replica
I20260810 18:11:30.473701 27741 raft_consensus.cc:2229] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:30.473917 26759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:30.474345 26759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:30.474345 27741 raft_consensus.cc:2258] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:30.476104 27739 tablet_replica.cc:330] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: stopping tablet replica
I20260810 18:11:30.476408 27739 raft_consensus.cc:2229] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:30.476612 27739 raft_consensus.cc:2258] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:30.478368 27741 ts_tablet_manager.cc:1905] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:30.479918 27491 tablet_service.cc:1467] Processing CreateTablet for tablet 48afd7c5abae4c409cf7d964ab91cb27 (DEFAULT_TABLE table=test-table [id=fd38830752464fa29957f1a4d72d4772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:30.480192 27491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48afd7c5abae4c409cf7d964ab91cb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:30.481918 27741 ts_tablet_manager.cc:1918] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:30.482097 27741 log.cc:1196] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/f86b88d02a1b4f4ca95c9265fd0aa147
I20260810 18:11:30.482530 27741 ts_tablet_manager.cc:1939] T f86b88d02a1b4f4ca95c9265fd0aa147 P 8ce490c82d5d45c3a19723f7c7239eb9: Deleting consensus metadata
I20260810 18:11:30.482599 27642 tablet_bootstrap.cc:492] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: Bootstrap starting.
I20260810 18:11:30.483624 27642 tablet_bootstrap.cc:654] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:30.484965 27642 tablet_bootstrap.cc:492] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: No bootstrap required, opened a new log
I20260810 18:11:30.484937 26741 catalog_manager.cc:4841] TS 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293): tablet f86b88d02a1b4f4ca95c9265fd0aa147 (table test-table [id=270bc59f617348f392cf1bc630249c79]) successfully deleted
I20260810 18:11:30.485069 27642 ts_tablet_manager.cc:1397] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 18:11:30.485570 27642 raft_consensus.cc:348] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.485687 27642 raft_consensus.cc:374] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:30.485716 27642 raft_consensus.cc:729] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364c7f10b5434ba691fb306d52b5cfae, State: Initialized, Role: FOLLOWER
I20260810 18:11:30.485836 27642 consensus_queue.cc:260] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.485970 27739 ts_tablet_manager.cc:1905] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:30.486716 27009 tablet_service.cc:1467] Processing CreateTablet for tablet 48afd7c5abae4c409cf7d964ab91cb27 (DEFAULT_TABLE table=test-table [id=fd38830752464fa29957f1a4d72d4772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:30.486979 27009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48afd7c5abae4c409cf7d964ab91cb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:30.487609 27740 tablet_replica.cc:330] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: stopping tablet replica
I20260810 18:11:30.487835 27204 tablet_service.cc:1467] Processing CreateTablet for tablet 48afd7c5abae4c409cf7d964ab91cb27 (DEFAULT_TABLE table=test-table [id=fd38830752464fa29957f1a4d72d4772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:30.487931 27740 raft_consensus.cc:2229] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:30.488088 27740 raft_consensus.cc:2258] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:30.488150 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48afd7c5abae4c409cf7d964ab91cb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:30.491072 27649 tablet_bootstrap.cc:492] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: Bootstrap starting.
I20260810 18:11:30.491374 27739 ts_tablet_manager.cc:1918] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:30.491467 27739 log.cc:1196] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/f86b88d02a1b4f4ca95c9265fd0aa147
I20260810 18:11:30.492082 27649 tablet_bootstrap.cc:654] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:30.492594 27739 ts_tablet_manager.cc:1939] T f86b88d02a1b4f4ca95c9265fd0aa147 P 364c7f10b5434ba691fb306d52b5cfae: Deleting consensus metadata
I20260810 18:11:30.494848 27635 tablet_bootstrap.cc:492] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: Bootstrap starting.
I20260810 18:11:30.495863 27635 tablet_bootstrap.cc:654] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:30.496524 27642 ts_tablet_manager.cc:1428] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20260810 18:11:30.497464 27649 tablet_bootstrap.cc:492] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: No bootstrap required, opened a new log
I20260810 18:11:30.497558 27649 ts_tablet_manager.cc:1397] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260810 18:11:30.497987 27649 raft_consensus.cc:348] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.498132 27649 raft_consensus.cc:374] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:30.498169 27649 raft_consensus.cc:729] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb96efa1156e4bb8af60d22470cc6c2a, State: Initialized, Role: FOLLOWER
I20260810 18:11:30.498288 27649 consensus_queue.cc:260] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.498541 27649 ts_tablet_manager.cc:1428] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:11:30.500566 26742 catalog_manager.cc:4841] TS 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865): tablet f86b88d02a1b4f4ca95c9265fd0aa147 (table test-table [id=270bc59f617348f392cf1bc630249c79]) successfully deleted
I20260810 18:11:30.501116 27740 ts_tablet_manager.cc:1905] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:30.501518 27635 tablet_bootstrap.cc:492] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: No bootstrap required, opened a new log
I20260810 18:11:30.501600 27635 ts_tablet_manager.cc:1397] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 18:11:30.502048 27635 raft_consensus.cc:348] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.502161 27635 raft_consensus.cc:374] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:30.502192 27635 raft_consensus.cc:729] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce490c82d5d45c3a19723f7c7239eb9, State: Initialized, Role: FOLLOWER
I20260810 18:11:30.502312 27635 consensus_queue.cc:260] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.502568 27635 ts_tablet_manager.cc:1428] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:30.504156 27740 ts_tablet_manager.cc:1918] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:30.504261 27740 log.cc:1196] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/f86b88d02a1b4f4ca95c9265fd0aa147
I20260810 18:11:30.504561 27740 ts_tablet_manager.cc:1939] T f86b88d02a1b4f4ca95c9265fd0aa147 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting consensus metadata
I20260810 18:11:30.505681 26741 catalog_manager.cc:4841] TS bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013): tablet f86b88d02a1b4f4ca95c9265fd0aa147 (table test-table [id=270bc59f617348f392cf1bc630249c79]) successfully deleted
I20260810 18:11:30.583233 27708 raft_consensus.cc:482] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:30.583413 27708 raft_consensus.cc:504] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.583956 27708 leader_election.cc:288] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293), 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865)
I20260810 18:11:30.584412 27047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48afd7c5abae4c409cf7d964ab91cb27" candidate_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" is_pre_election: true
I20260810 18:11:30.584443 27518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48afd7c5abae4c409cf7d964ab91cb27" candidate_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364c7f10b5434ba691fb306d52b5cfae" is_pre_election: true
I20260810 18:11:30.584570 27047 raft_consensus.cc:2454] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb96efa1156e4bb8af60d22470cc6c2a in term 0.
I20260810 18:11:30.584604 27518 raft_consensus.cc:2454] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb96efa1156e4bb8af60d22470cc6c2a in term 0.
I20260810 18:11:30.584998 27161 leader_election.cc:302] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ce490c82d5d45c3a19723f7c7239eb9, bb96efa1156e4bb8af60d22470cc6c2a; no voters:
I20260810 18:11:30.585245 27708 raft_consensus.cc:2781] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:30.585337 27708 raft_consensus.cc:482] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:30.585381 27708 raft_consensus.cc:3037] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:30.586799 27708 raft_consensus.cc:504] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.587204 27708 leader_election.cc:288] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [CANDIDATE]: Term 1 election: Requested vote from peers 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293), 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865)
I20260810 18:11:30.587553 27047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48afd7c5abae4c409cf7d964ab91cb27" candidate_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce490c82d5d45c3a19723f7c7239eb9"
I20260810 18:11:30.587707 27047 raft_consensus.cc:3037] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:30.587711 27518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48afd7c5abae4c409cf7d964ab91cb27" candidate_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364c7f10b5434ba691fb306d52b5cfae"
I20260810 18:11:30.587833 27518 raft_consensus.cc:3037] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:30.588881 27047 raft_consensus.cc:2454] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb96efa1156e4bb8af60d22470cc6c2a in term 1.
I20260810 18:11:30.589184 27161 leader_election.cc:302] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ce490c82d5d45c3a19723f7c7239eb9, bb96efa1156e4bb8af60d22470cc6c2a; no voters:
I20260810 18:11:30.589411 27518 raft_consensus.cc:2454] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb96efa1156e4bb8af60d22470cc6c2a in term 1.
I20260810 18:11:30.589643 27708 raft_consensus.cc:2781] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:30.589772 27708 raft_consensus.cc:686] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 LEADER]: Becoming Leader. State: Replica: bb96efa1156e4bb8af60d22470cc6c2a, State: Running, Role: LEADER
I20260810 18:11:30.589920 27708 consensus_queue.cc:237] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:30.591791 26759 catalog_manager.cc:5495] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66). New cstate: current_term: 1 leader_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } health_report { overall_health: UNKNOWN } } }
I20260810 18:11:30.626112 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.622783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.23.65.66:43013)
0810 18:11:30.623068 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.623071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.623080 (+ 9us) server_negotiation.cc:407] Connection header received
0810 18:11:30.623114 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.623116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.623153 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.623204 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:30.623500 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.624015 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.625165 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.625387 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.625455 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.625710 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.625774 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.625822 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.625933 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.625981 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260810 18:11:30.635262 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.632024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:43013 (local address 127.23.65.66:60351)
0810 18:11:30.632186 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:30.632198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:30.632274 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:30.632495 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:30.632497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:30.632502 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:30.632594 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.632598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.633527 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.633530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.634068 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:30.634075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.634635 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.634637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:30.634703 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.634729 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.634992 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.634996 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:11:30.635037 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 18:11:30.635104 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":15}
I20260810 18:11:30.635289 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:30.632109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:60351 (local address 127.23.65.66:43013)
0810 18:11:30.632226 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:30.632230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:30.632246 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:11:30.632363 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:30.632366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:30.632414 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:30.632492 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:30.632685 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.633424 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.634187 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:30.634524 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:30.634638 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:30.634839 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:30.634917 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:30.634995 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:30.635057 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 18:11:30.635119 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":44}
I20260810 18:11:30.640810 27047 raft_consensus.cc:1261] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Refusing update from remote peer bb96efa1156e4bb8af60d22470cc6c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:30.640901 27519 raft_consensus.cc:1261] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Refusing update from remote peer bb96efa1156e4bb8af60d22470cc6c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:30.641343 27664 consensus_queue.cc:1035] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:30.641518 27694 consensus_queue.cc:1035] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:30.644038 27703 mvcc.cc:205] Tried to move back new op lower bound from 7317034969660289024 to 7317034969457913856. Current Snapshot: MvccSnapshot[applied={T|T < 7317034969660289024}]
W20260810 18:11:30.656621 27703 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 18:11:32.209781 26760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:32.209940 26760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:32.212502 26760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb: Sending DeleteTablet for 3 replicas of tablet 48afd7c5abae4c409cf7d964ab91cb27
I20260810 18:11:32.213039 27009 tablet_service.cc:1514] Processing DeleteTablet for tablet 48afd7c5abae4c409cf7d964ab91cb27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:32 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:51872
I20260810 18:11:32.213124 27491 tablet_service.cc:1514] Processing DeleteTablet for tablet 48afd7c5abae4c409cf7d964ab91cb27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:32 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:51080
I20260810 18:11:32.213416 27789 tablet_replica.cc:330] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: stopping tablet replica
I20260810 18:11:32.213452 27242 tablet_service.cc:1514] Processing DeleteTablet for tablet 48afd7c5abae4c409cf7d964ab91cb27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:32 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:53726
I20260810 18:11:32.213555 27789 raft_consensus.cc:2229] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:32.213582 27793 tablet_replica.cc:330] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: stopping tablet replica
I20260810 18:11:32.213675 27789 raft_consensus.cc:2258] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:32.213788 27793 raft_consensus.cc:2229] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:32.213904 27793 raft_consensus.cc:2258] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:32.214351 27789 ts_tablet_manager.cc:1905] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:32.214442 27793 ts_tablet_manager.cc:1905] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:32.217043 27789 ts_tablet_manager.cc:1918] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:11:32.217044 27793 ts_tablet_manager.cc:1918] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:11:32.217131 27789 log.cc:1196] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/48afd7c5abae4c409cf7d964ab91cb27
I20260810 18:11:32.217131 27793 log.cc:1196] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/48afd7c5abae4c409cf7d964ab91cb27
I20260810 18:11:32.217399 27793 ts_tablet_manager.cc:1939] T 48afd7c5abae4c409cf7d964ab91cb27 P 364c7f10b5434ba691fb306d52b5cfae: Deleting consensus metadata
I20260810 18:11:32.217399 27789 ts_tablet_manager.cc:1939] T 48afd7c5abae4c409cf7d964ab91cb27 P 8ce490c82d5d45c3a19723f7c7239eb9: Deleting consensus metadata
I20260810 18:11:32.218143 26741 catalog_manager.cc:4841] TS 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293): tablet 48afd7c5abae4c409cf7d964ab91cb27 (table test-table [id=fd38830752464fa29957f1a4d72d4772]) successfully deleted
I20260810 18:11:32.218262 26742 catalog_manager.cc:4841] TS 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865): tablet 48afd7c5abae4c409cf7d964ab91cb27 (table test-table [id=fd38830752464fa29957f1a4d72d4772]) successfully deleted
I20260810 18:11:32.219739 27794 tablet_replica.cc:330] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: stopping tablet replica
I20260810 18:11:32.219882 27794 raft_consensus.cc:2229] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:32.220228 27794 raft_consensus.cc:2258] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:32.220861 27794 ts_tablet_manager.cc:1905] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:32.223757 27794 ts_tablet_manager.cc:1918] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:11:32.223824 27794 log.cc:1196] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/48afd7c5abae4c409cf7d964ab91cb27
I20260810 18:11:32.224052 27794 ts_tablet_manager.cc:1939] T 48afd7c5abae4c409cf7d964ab91cb27 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting consensus metadata
I20260810 18:11:32.224713 26741 catalog_manager.cc:4841] TS bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013): tablet 48afd7c5abae4c409cf7d964ab91cb27 (table test-table [id=fd38830752464fa29957f1a4d72d4772]) successfully deleted
Aug 10 18:11:32 dist-test-slave-jpj1 krb5kdc[26692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385492, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:11:32 dist-test-slave-jpj1 krb5kdc[26692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385492, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:32.245843 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.236753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.23.65.126:35411)
0810 18:11:32.237398 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:32.237435 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:32.237450 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:32.237489 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:32.237491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:32.237531 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:32.237611 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:32.238391 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.238995 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.239919 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.240156 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.240248 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.242727 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:32.242731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:32.242734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:32.242767 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:32.244246 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:32.244656 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:32.244660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:32.244661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:32.244743 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:32.245001 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:32.245005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:32.245007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:32.245193 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:32.245322 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:32.245552 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 18:11:32.245676 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":105,"threads_started":1}
I20260810 18:11:32.248354 26760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:32.248756 26760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:32.253499 27009 tablet_service.cc:1467] Processing CreateTablet for tablet 96fa6f00e1b942d2ab915719b4b9c917 (DEFAULT_TABLE table=test-table [id=ef5d1ace1d6b4a8caf6500dd02d2b8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:32.253566 27242 tablet_service.cc:1467] Processing CreateTablet for tablet 96fa6f00e1b942d2ab915719b4b9c917 (DEFAULT_TABLE table=test-table [id=ef5d1ace1d6b4a8caf6500dd02d2b8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:32.253718 27491 tablet_service.cc:1467] Processing CreateTablet for tablet 96fa6f00e1b942d2ab915719b4b9c917 (DEFAULT_TABLE table=test-table [id=ef5d1ace1d6b4a8caf6500dd02d2b8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:32.253863 27009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fa6f00e1b942d2ab915719b4b9c917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:32.253901 27242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fa6f00e1b942d2ab915719b4b9c917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:32.253988 27491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fa6f00e1b942d2ab915719b4b9c917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:32.256695 27811 tablet_bootstrap.cc:492] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: Bootstrap starting.
I20260810 18:11:32.256713 27810 tablet_bootstrap.cc:492] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: Bootstrap starting.
I20260810 18:11:32.257536 27810 tablet_bootstrap.cc:654] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:32.258324 27811 tablet_bootstrap.cc:654] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:32.258683 27810 tablet_bootstrap.cc:492] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: No bootstrap required, opened a new log
I20260810 18:11:32.258767 27810 ts_tablet_manager.cc:1397] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:32.259169 27810 raft_consensus.cc:348] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.259305 27810 raft_consensus.cc:374] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:32.259356 27810 raft_consensus.cc:729] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb96efa1156e4bb8af60d22470cc6c2a, State: Initialized, Role: FOLLOWER
I20260810 18:11:32.259478 27810 consensus_queue.cc:260] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.259728 27816 tablet_bootstrap.cc:492] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: Bootstrap starting.
I20260810 18:11:32.259855 27810 ts_tablet_manager.cc:1428] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:32.260426 27811 tablet_bootstrap.cc:492] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: No bootstrap required, opened a new log
I20260810 18:11:32.260511 27811 ts_tablet_manager.cc:1397] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 18:11:32.260855 27816 tablet_bootstrap.cc:654] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:32.260890 27811 raft_consensus.cc:348] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.260999 27811 raft_consensus.cc:374] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:32.261037 27811 raft_consensus.cc:729] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364c7f10b5434ba691fb306d52b5cfae, State: Initialized, Role: FOLLOWER
I20260810 18:11:32.261161 27811 consensus_queue.cc:260] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.261559 27811 ts_tablet_manager.cc:1428] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:32.262454 27816 tablet_bootstrap.cc:492] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: No bootstrap required, opened a new log
I20260810 18:11:32.262555 27816 ts_tablet_manager.cc:1397] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:11:32.263113 27816 raft_consensus.cc:348] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.263228 27816 raft_consensus.cc:374] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:32.263262 27816 raft_consensus.cc:729] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce490c82d5d45c3a19723f7c7239eb9, State: Initialized, Role: FOLLOWER
I20260810 18:11:32.263396 27816 consensus_queue.cc:260] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.263803 27816 ts_tablet_manager.cc:1428] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 18:11:32.376756 27820 raft_consensus.cc:482] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:32.376910 27820 raft_consensus.cc:504] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.378046 27820 leader_election.cc:288] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293), bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013)
I20260810 18:11:32.382928 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.378401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:33521 (local address 127.23.65.65:38293)
0810 18:11:32.378757 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:32.378766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:32.378786 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:11:32.378958 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:32.378962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:32.379022 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:32.379105 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:32.379590 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.380439 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.381822 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.382353 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.382472 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.382529 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:32.382706 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 18:11:32.382786 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260810 18:11:32.382930 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.378293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:38293 (local address 127.23.65.67:33521)
0810 18:11:32.378684 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:32.378700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:32.378805 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:32.379257 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:32.379260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:32.379281 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:32.379435 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:32.379443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.380563 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.380566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:32.381667 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:32.381678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.382481 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.382484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:32.382581 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.382608 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:11:32.382689 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 18:11:32.382759 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":204,"threads_started":1}
I20260810 18:11:32.383571 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.378831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:43013 (local address 127.23.65.67:39617)
0810 18:11:32.379341 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:32.379354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:32.379467 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:32.379968 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:32.379971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:32.379991 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:32.380090 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:32.380095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.381019 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.381023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:32.382280 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:32.382292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.383104 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.383108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:32.383269 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.383290 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 18:11:32.383342 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 18:11:32.383411 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":183,"threads_started":1}
I20260810 18:11:32.383749 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fa6f00e1b942d2ab915719b4b9c917" candidate_uuid: "364c7f10b5434ba691fb306d52b5cfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" is_pre_election: true
I20260810 18:11:32.383920 27046 raft_consensus.cc:2454] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 364c7f10b5434ba691fb306d52b5cfae in term 0.
I20260810 18:11:32.384132 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.379004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:39617 (local address 127.23.65.66:43013)
0810 18:11:32.379724 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:32.379751 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:32.379765 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:32.379808 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:32.379811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:32.379857 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:32.379932 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:32.380200 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.380869 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.382424 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.382961 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.383771 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.383835 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:32.383871 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 18:11:32.383943 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":123,"threads_started":1}
I20260810 18:11:32.384420 27398 leader_election.cc:302] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c7f10b5434ba691fb306d52b5cfae, 8ce490c82d5d45c3a19723f7c7239eb9; no voters:
I20260810 18:11:32.384567 27272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fa6f00e1b942d2ab915719b4b9c917" candidate_uuid: "364c7f10b5434ba691fb306d52b5cfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" is_pre_election: true
I20260810 18:11:32.384668 27820 raft_consensus.cc:2781] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:32.384727 27820 raft_consensus.cc:482] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:32.384725 27272 raft_consensus.cc:2454] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 364c7f10b5434ba691fb306d52b5cfae in term 0.
I20260810 18:11:32.384763 27820 raft_consensus.cc:3037] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:32.386178 27820 raft_consensus.cc:504] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.386492 27820 leader_election.cc:288] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [CANDIDATE]: Term 1 election: Requested vote from peers 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293), bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013)
I20260810 18:11:32.386801 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fa6f00e1b942d2ab915719b4b9c917" candidate_uuid: "364c7f10b5434ba691fb306d52b5cfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce490c82d5d45c3a19723f7c7239eb9"
I20260810 18:11:32.386925 27046 raft_consensus.cc:3037] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:32.386940 27272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fa6f00e1b942d2ab915719b4b9c917" candidate_uuid: "364c7f10b5434ba691fb306d52b5cfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb96efa1156e4bb8af60d22470cc6c2a"
I20260810 18:11:32.387056 27272 raft_consensus.cc:3037] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:32.388159 27046 raft_consensus.cc:2454] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 364c7f10b5434ba691fb306d52b5cfae in term 1.
I20260810 18:11:32.388163 27272 raft_consensus.cc:2454] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 364c7f10b5434ba691fb306d52b5cfae in term 1.
I20260810 18:11:32.388429 27398 leader_election.cc:302] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c7f10b5434ba691fb306d52b5cfae, 8ce490c82d5d45c3a19723f7c7239eb9; no voters:
I20260810 18:11:32.388679 27820 raft_consensus.cc:2781] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:32.388899 27820 raft_consensus.cc:686] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 1 LEADER]: Becoming Leader. State: Replica: 364c7f10b5434ba691fb306d52b5cfae, State: Running, Role: LEADER
I20260810 18:11:32.389142 27820 consensus_queue.cc:237] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.391708 26759 catalog_manager.cc:5495] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae reported cstate change: term changed from 0 to 1, leader changed from <none> to 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67). New cstate: current_term: 1 leader_uuid: "364c7f10b5434ba691fb306d52b5cfae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } health_report { overall_health: HEALTHY } } }
I20260810 18:11:32.425176 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.421002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.23.65.67:37865)
0810 18:11:32.421340 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:32.421347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:32.421359 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:11:32.421433 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:32.421435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:32.421487 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:32.421587 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:32.421863 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.422414 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.423688 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.423942 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.424023 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.424299 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:32.424754 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:32.424838 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:32.424932 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 18:11:32.424981 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":164,"threads_started":1}
I20260810 18:11:32.436820 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.433035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:41819 (local address 127.23.65.66:43013)
0810 18:11:32.433160 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:32.433165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:32.433436 (+ 271us) server_negotiation.cc:407] Connection header received
0810 18:11:32.433565 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:32.433568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:32.433618 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:32.433678 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:32.434038 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.434577 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.435746 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.435961 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.436048 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.436383 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:32.436462 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:32.436521 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:32.436618 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 18:11:32.436674 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:11:32.436904 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.432826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:43013 (local address 127.23.65.67:41819)
0810 18:11:32.433349 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:32.433363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:32.433467 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:32.433696 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:32.433706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:32.433713 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:32.433926 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:32.433933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.434677 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.434680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:32.435628 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:32.435638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.436058 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.436061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:32.436205 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.436251 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:32.436538 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:32.436545 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:11:32.436600 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 18:11:32.436684 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":100,"threads_started":1}
I20260810 18:11:32.443935 27046 raft_consensus.cc:1261] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Refusing update from remote peer 364c7f10b5434ba691fb306d52b5cfae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:32.443929 27272 raft_consensus.cc:1261] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Refusing update from remote peer 364c7f10b5434ba691fb306d52b5cfae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:32.444456 27833 consensus_queue.cc:1035] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:32.444684 27820 consensus_queue.cc:1035] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:32.448956 27759 mvcc.cc:205] Tried to move back new op lower bound from 7317034977045569536 to 7317034976828190720. Current Snapshot: MvccSnapshot[applied={T|T < 7317034977045569536}]
I20260810 18:11:32.466005 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.461564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:37865 (local address 127.23.65.66:44845)
0810 18:11:32.462052 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:32.462068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:32.462184 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:32.462506 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:32.462509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:32.462518 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:32.462630 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:32.462637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.463462 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.463465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:32.464188 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:32.464200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.465228 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.465232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:32.465322 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.465356 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:32.465697 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:32.465706 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:11:32.465772 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:11:32.465865 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":170,"threads_started":1}
I20260810 18:11:32.465992 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:32.461953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:44845 (local address 127.23.65.67:37865)
0810 18:11:32.462050 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:32.462054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:32.462149 (+ 95us) server_negotiation.cc:407] Connection header received
0810 18:11:32.462273 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:32.462276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:32.462320 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:32.462382 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:32.462764 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.463344 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.464471 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:32.464939 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:32.465049 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:32.465513 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:32.465612 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:32.465691 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:32.465781 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 18:11:32.465843 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:11:32.488845 26759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:32.489038 26759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:32.491497 26759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb: Sending DeleteTablet for 3 replicas of tablet 96fa6f00e1b942d2ab915719b4b9c917
I20260810 18:11:32.492290 27009 tablet_service.cc:1514] Processing DeleteTablet for tablet 96fa6f00e1b942d2ab915719b4b9c917 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:32 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:51872
I20260810 18:11:32.492506 27789 tablet_replica.cc:330] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: stopping tablet replica
I20260810 18:11:32.492645 27789 raft_consensus.cc:2229] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:32.492713 27491 tablet_service.cc:1514] Processing DeleteTablet for tablet 96fa6f00e1b942d2ab915719b4b9c917 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:32 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:51080
I20260810 18:11:32.492782 27789 raft_consensus.cc:2258] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:32.492895 27793 tablet_replica.cc:330] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: stopping tablet replica
I20260810 18:11:32.492996 27793 raft_consensus.cc:2229] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:32.493208 27242 tablet_service.cc:1514] Processing DeleteTablet for tablet 96fa6f00e1b942d2ab915719b4b9c917 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:32 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:53726
I20260810 18:11:32.493285 27793 raft_consensus.cc:2258] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:32.493433 27794 tablet_replica.cc:330] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: stopping tablet replica
I20260810 18:11:32.493554 27794 raft_consensus.cc:2229] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:32.493669 27794 raft_consensus.cc:2258] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:32.494040 27793 ts_tablet_manager.cc:1905] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:32.494463 27794 ts_tablet_manager.cc:1905] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:32.497125 27794 ts_tablet_manager.cc:1918] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:32.497145 27793 ts_tablet_manager.cc:1918] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:32.497211 27793 log.cc:1196] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/96fa6f00e1b942d2ab915719b4b9c917
I20260810 18:11:32.497211 27794 log.cc:1196] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/96fa6f00e1b942d2ab915719b4b9c917
I20260810 18:11:32.497473 27794 ts_tablet_manager.cc:1939] T 96fa6f00e1b942d2ab915719b4b9c917 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting consensus metadata
I20260810 18:11:32.497473 27793 ts_tablet_manager.cc:1939] T 96fa6f00e1b942d2ab915719b4b9c917 P 364c7f10b5434ba691fb306d52b5cfae: Deleting consensus metadata
I20260810 18:11:32.498104 26742 catalog_manager.cc:4841] TS 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865): tablet 96fa6f00e1b942d2ab915719b4b9c917 (table test-table [id=ef5d1ace1d6b4a8caf6500dd02d2b8ee]) successfully deleted
I20260810 18:11:32.498104 26741 catalog_manager.cc:4841] TS bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013): tablet 96fa6f00e1b942d2ab915719b4b9c917 (table test-table [id=ef5d1ace1d6b4a8caf6500dd02d2b8ee]) successfully deleted
I20260810 18:11:32.498912 27789 ts_tablet_manager.cc:1905] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:32.499296 26759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:32.499751 26759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:32.501504 27789 ts_tablet_manager.cc:1918] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:32.501575 27789 log.cc:1196] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/96fa6f00e1b942d2ab915719b4b9c917
I20260810 18:11:32.501782 27789 ts_tablet_manager.cc:1939] T 96fa6f00e1b942d2ab915719b4b9c917 P 8ce490c82d5d45c3a19723f7c7239eb9: Deleting consensus metadata
I20260810 18:11:32.507220 27242 tablet_service.cc:1467] Processing CreateTablet for tablet 80d6fbf0d34a474ea95167356ad7b771 (DEFAULT_TABLE table=test-table [id=11ce3dff4a8d49af97a560560ddb6d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:32.507433 27491 tablet_service.cc:1467] Processing CreateTablet for tablet 80d6fbf0d34a474ea95167356ad7b771 (DEFAULT_TABLE table=test-table [id=11ce3dff4a8d49af97a560560ddb6d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:32.507715 27491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d6fbf0d34a474ea95167356ad7b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:32.507777 27009 tablet_service.cc:1467] Processing CreateTablet for tablet 80d6fbf0d34a474ea95167356ad7b771 (DEFAULT_TABLE table=test-table [id=11ce3dff4a8d49af97a560560ddb6d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:32.508039 27009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d6fbf0d34a474ea95167356ad7b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:32.509970 27816 tablet_bootstrap.cc:492] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9: Bootstrap starting.
I20260810 18:11:32.510257 27811 tablet_bootstrap.cc:492] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: Bootstrap starting.
I20260810 18:11:32.510779 27816 tablet_bootstrap.cc:654] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:32.507513 27242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d6fbf0d34a474ea95167356ad7b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:32.511919 27811 tablet_bootstrap.cc:654] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:32.512189 27816 tablet_bootstrap.cc:492] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9: No bootstrap required, opened a new log
I20260810 18:11:32.512292 27816 ts_tablet_manager.cc:1397] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:32.512933 27816 raft_consensus.cc:348] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.513055 27816 raft_consensus.cc:374] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:32.513087 27816 raft_consensus.cc:729] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce490c82d5d45c3a19723f7c7239eb9, State: Initialized, Role: FOLLOWER
I20260810 18:11:32.513216 27816 consensus_queue.cc:260] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.513471 27816 ts_tablet_manager.cc:1428] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:11:32.514613 27810 tablet_bootstrap.cc:492] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: Bootstrap starting.
I20260810 18:11:32.515482 27810 tablet_bootstrap.cc:654] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:32.517053 27810 tablet_bootstrap.cc:492] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: No bootstrap required, opened a new log
I20260810 18:11:32.517148 27810 ts_tablet_manager.cc:1397] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:11:32.517529 27810 raft_consensus.cc:348] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.517655 27810 raft_consensus.cc:374] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:32.517694 27810 raft_consensus.cc:729] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb96efa1156e4bb8af60d22470cc6c2a, State: Initialized, Role: FOLLOWER
I20260810 18:11:32.517822 27810 consensus_queue.cc:260] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.518085 27810 ts_tablet_manager.cc:1428] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:11:32.520217 26741 catalog_manager.cc:4841] TS 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293): tablet 96fa6f00e1b942d2ab915719b4b9c917 (table test-table [id=ef5d1ace1d6b4a8caf6500dd02d2b8ee]) successfully deleted
I20260810 18:11:32.521123 27811 tablet_bootstrap.cc:492] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: No bootstrap required, opened a new log
I20260810 18:11:32.521232 27811 ts_tablet_manager.cc:1397] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20260810 18:11:32.521922 27811 raft_consensus.cc:348] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.522039 27811 raft_consensus.cc:374] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:32.522073 27811 raft_consensus.cc:729] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364c7f10b5434ba691fb306d52b5cfae, State: Initialized, Role: FOLLOWER
I20260810 18:11:32.522205 27811 consensus_queue.cc:260] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.522503 27811 ts_tablet_manager.cc:1428] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:11:32.573570 27845 raft_consensus.cc:482] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:32.573732 27845 raft_consensus.cc:504] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.574187 27845 leader_election.cc:288] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293), bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013)
I20260810 18:11:32.574679 27272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d6fbf0d34a474ea95167356ad7b771" candidate_uuid: "364c7f10b5434ba691fb306d52b5cfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" is_pre_election: true
I20260810 18:11:32.574704 27047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d6fbf0d34a474ea95167356ad7b771" candidate_uuid: "364c7f10b5434ba691fb306d52b5cfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" is_pre_election: true
I20260810 18:11:32.574826 27272 raft_consensus.cc:2454] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 364c7f10b5434ba691fb306d52b5cfae in term 0.
I20260810 18:11:32.574873 27047 raft_consensus.cc:2454] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 364c7f10b5434ba691fb306d52b5cfae in term 0.
I20260810 18:11:32.575153 27398 leader_election.cc:302] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c7f10b5434ba691fb306d52b5cfae, bb96efa1156e4bb8af60d22470cc6c2a; no voters:
I20260810 18:11:32.575338 27845 raft_consensus.cc:2781] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:32.575402 27845 raft_consensus.cc:482] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:32.575439 27845 raft_consensus.cc:3037] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:32.576769 27845 raft_consensus.cc:504] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.577068 27845 leader_election.cc:288] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [CANDIDATE]: Term 1 election: Requested vote from peers 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293), bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013)
I20260810 18:11:32.577466 27272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d6fbf0d34a474ea95167356ad7b771" candidate_uuid: "364c7f10b5434ba691fb306d52b5cfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb96efa1156e4bb8af60d22470cc6c2a"
I20260810 18:11:32.577466 27047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d6fbf0d34a474ea95167356ad7b771" candidate_uuid: "364c7f10b5434ba691fb306d52b5cfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce490c82d5d45c3a19723f7c7239eb9"
I20260810 18:11:32.577579 27272 raft_consensus.cc:3037] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:32.577589 27047 raft_consensus.cc:3037] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:32.578492 27272 raft_consensus.cc:2454] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 364c7f10b5434ba691fb306d52b5cfae in term 1.
I20260810 18:11:32.578773 27398 leader_election.cc:302] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c7f10b5434ba691fb306d52b5cfae, bb96efa1156e4bb8af60d22470cc6c2a; no voters:
I20260810 18:11:32.578933 27047 raft_consensus.cc:2454] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 364c7f10b5434ba691fb306d52b5cfae in term 1.
I20260810 18:11:32.578969 27845 raft_consensus.cc:2781] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:32.579041 27845 raft_consensus.cc:686] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 1 LEADER]: Becoming Leader. State: Replica: 364c7f10b5434ba691fb306d52b5cfae, State: Running, Role: LEADER
I20260810 18:11:32.579185 27845 consensus_queue.cc:237] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } }
I20260810 18:11:32.581084 26759 catalog_manager.cc:5495] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae reported cstate change: term changed from 0 to 1, leader changed from <none> to 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67). New cstate: current_term: 1 leader_uuid: "364c7f10b5434ba691fb306d52b5cfae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "364c7f10b5434ba691fb306d52b5cfae" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 37865 } health_report { overall_health: HEALTHY } } }
I20260810 18:11:32.598151 27046 raft_consensus.cc:1261] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9 [term 1 FOLLOWER]: Refusing update from remote peer 364c7f10b5434ba691fb306d52b5cfae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:32.598831 27272 raft_consensus.cc:1261] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Refusing update from remote peer 364c7f10b5434ba691fb306d52b5cfae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:32.599042 27845 consensus_queue.cc:1035] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce490c82d5d45c3a19723f7c7239eb9" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:32.599232 27820 consensus_queue.cc:1035] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb96efa1156e4bb8af60d22470cc6c2a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 43013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:32.604885 27764 mvcc.cc:205] Tried to move back new op lower bound from 7317034977678807040 to 7317034977606012928. Current Snapshot: MvccSnapshot[applied={T|T < 7317034977678807040}]
W20260810 18:11:32.614753 27759 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 18:11:34.165705 26760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:34.165866 26760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:34.168169 26760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de5d536782047a28392eaf84bd34adb: Sending DeleteTablet for 3 replicas of tablet 80d6fbf0d34a474ea95167356ad7b771
I20260810 18:11:34.168648 27009 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d6fbf0d34a474ea95167356ad7b771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:34 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:51872
I20260810 18:11:34.168730 27242 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d6fbf0d34a474ea95167356ad7b771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:34 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:53726
I20260810 18:11:34.169023 27896 tablet_replica.cc:330] T 80d6fbf0d34a474ea95167356ad7b771 P 8ce490c82d5d45c3a19723f7c7239eb9: stopping tablet replica
I20260810 18:11:34.169013 27491 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d6fbf0d34a474ea95167356ad7b771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:34 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:51080
I20260810 18:11:34.169121 27898 tablet_replica.cc:330] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: stopping tablet replica
I20260810 18:11:34.169154 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 26893
I20260810 18:11:34.169277 27898 raft_consensus.cc:2229] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:34.169377 27898 raft_consensus.cc:2258] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:34.186722 27898 ts_tablet_manager.cc:1905] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:34.189505 27899 tablet_replica.cc:330] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: stopping tablet replica
I20260810 18:11:34.189685 27899 raft_consensus.cc:2229] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:34.189770 27898 ts_tablet_manager.cc:1918] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:11:34.189865 27898 log.cc:1196] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/80d6fbf0d34a474ea95167356ad7b771
I20260810 18:11:34.189908 27899 raft_consensus.cc:2258] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:34.190155 27898 ts_tablet_manager.cc:1939] T 80d6fbf0d34a474ea95167356ad7b771 P bb96efa1156e4bb8af60d22470cc6c2a: Deleting consensus metadata
I20260810 18:11:34.190557 27899 ts_tablet_manager.cc:1905] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:34.191017 26741 catalog_manager.cc:4841] TS bb96efa1156e4bb8af60d22470cc6c2a (127.23.65.66:43013): tablet 80d6fbf0d34a474ea95167356ad7b771 (table test-table [id=11ce3dff4a8d49af97a560560ddb6d0a]) successfully deleted
I20260810 18:11:34.193176 27899 ts_tablet_manager.cc:1918] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:11:34.193243 27899 log.cc:1196] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/80d6fbf0d34a474ea95167356ad7b771
I20260810 18:11:34.193490 27899 ts_tablet_manager.cc:1939] T 80d6fbf0d34a474ea95167356ad7b771 P 364c7f10b5434ba691fb306d52b5cfae: Deleting consensus metadata
I20260810 18:11:34.194330 26742 catalog_manager.cc:4841] TS 364c7f10b5434ba691fb306d52b5cfae (127.23.65.67:37865): tablet 80d6fbf0d34a474ea95167356ad7b771 (table test-table [id=11ce3dff4a8d49af97a560560ddb6d0a]) successfully deleted
W20260810 18:11:34.195195 26741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:38293 (error 108)
I20260810 18:11:34.195384 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 27124
I20260810 18:11:34.203920 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:34.200344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:38293 (local address 127.0.0.1:51304)
0810 18:11:34.203743 (+ 3399us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:34.203822 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3320,"thread_start_us":88,"threads_started":1}
W20260810 18:11:34.204085 26741 catalog_manager.cc:4584] TS 8ce490c82d5d45c3a19723f7c7239eb9 (127.23.65.65:38293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80d6fbf0d34a474ea95167356ad7b771: Network error: Client connection negotiation failed: client connection to 127.23.65.65:38293: connect: Connection refused (error 111)
I20260810 18:11:34.208896 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 27370
I20260810 18:11:34.220201 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 26714
2026-08-10T18:11:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6477 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:34 dist-test-slave-jpj1 krb5kdc[27905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:34 dist-test-slave-jpj1 krb5kdc[27905](info): set up 2 sockets
Aug 10 18:11:34 dist-test-slave-jpj1 krb5kdc[27905](info): commencing operation
krb5kdc: starting...
W20260810 18:11:35.373772 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:35 dist-test-slave-jpj1 krb5kdc[27905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:35.496738 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40595
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:40595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:35.604704 27930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:35.604933 27930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:35.604969 27930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:35.608763 27930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:35.608807 27930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:35.608824 27930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:35.608844 27930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:35.608860 27930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:35.613853 27930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:40595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:35.614742 27930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:35.615649 27930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:35.623792 27937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:35.623853 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:35.623975 27930 server_base.cc:1034] running on GCE node
W20260810 18:11:35.623867 27938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:35.624732 27930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:35.625808 27930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:35.626993 27930 hybrid_clock.cc:648] HybridClock initialized: now 1786385495626978 us; error 26 us; skew 500 ppm
Aug 10 18:11:35 dist-test-slave-jpj1 krb5kdc[27905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385495, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:35.629828 27930 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:35.630760 27930 webserver.cc:460] Webserver started at http://127.23.65.126:37661/ using document root <none> and password file <none>
I20260810 18:11:35.631294 27930 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:35.631361 27930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:35.631596 27930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:35.633628 27930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "7c98b1c3358949e5affc3e8acdad852a"
format_stamp: "Formatted at 2026-08-10 18:11:35 on dist-test-slave-jpj1"
server_key: "66295e836b2420e08723eea761be44bc"
server_key_iv: "857a5621b043399f06ca35393c029082"
server_key_version: "encryptionkey@0"
I20260810 18:11:35.634181 27930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "7c98b1c3358949e5affc3e8acdad852a"
format_stamp: "Formatted at 2026-08-10 18:11:35 on dist-test-slave-jpj1"
server_key: "66295e836b2420e08723eea761be44bc"
server_key_iv: "857a5621b043399f06ca35393c029082"
server_key_version: "encryptionkey@0"
I20260810 18:11:35.637910 27930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 18:11:35.640303 27950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:35.641445 27930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:11:35.641562 27930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "7c98b1c3358949e5affc3e8acdad852a"
format_stamp: "Formatted at 2026-08-10 18:11:35 on dist-test-slave-jpj1"
server_key: "66295e836b2420e08723eea761be44bc"
server_key_iv: "857a5621b043399f06ca35393c029082"
server_key_version: "encryptionkey@0"
I20260810 18:11:35.641656 27930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:35.656314 27930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:35.658927 27930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:35.659097 27930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:35.665946 27930 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:40595
I20260810 18:11:35.665970 28036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:40595 every 8 connection(s)
I20260810 18:11:35.666857 27930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:35.669564 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:35.672219 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 27930
I20260810 18:11:35.672343 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:35.672624 23813 external_mini_cluster.cc:1428] Setting key 4c0374a9410e0acaad09c48d4b946e96
I20260810 18:11:35.675429 28037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a: Bootstrap starting.
I20260810 18:11:35.677870 28037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:35.678756 28037 log.cc:824] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:35.680624 28037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a: No bootstrap required, opened a new log
Aug 10 18:11:35 dist-test-slave-jpj1 krb5kdc[27905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:35.684456 28037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c98b1c3358949e5affc3e8acdad852a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40595 } }
I20260810 18:11:35.684728 28037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:35.684801 28037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c98b1c3358949e5affc3e8acdad852a, State: Initialized, Role: FOLLOWER
I20260810 18:11:35.685297 28037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c98b1c3358949e5affc3e8acdad852a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40595 } }
I20260810 18:11:35.685456 28037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:35.685527 28037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:35.685624 28037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:35.686681 28037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c98b1c3358949e5affc3e8acdad852a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40595 } }
I20260810 18:11:35.687044 28037 leader_election.cc:302] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c98b1c3358949e5affc3e8acdad852a; no voters:
I20260810 18:11:35.687361 28037 leader_election.cc:288] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:35.687454 28043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:35.687698 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:35.673942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.23.65.126:40595)
0810 18:11:35.674349 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:35.674363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:35.674400 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:11:35.675001 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:35.675016 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:35.675337 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:35.675662 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:35.676499 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:35.677502 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:35.678691 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:35.679208 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:35.679366 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:35.681937 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:35.681943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:35.681960 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:35.681991 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:35.684517 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:35.684984 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:35.684990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:35.684996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:35.685143 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:35.685394 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:35.685397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:35.685399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:35.685919 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:35.686125 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:35.686343 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 18:11:35.686646 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260810 18:11:35.687696 28043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [term 1 LEADER]: Becoming Leader. State: Replica: 7c98b1c3358949e5affc3e8acdad852a, State: Running, Role: LEADER
I20260810 18:11:35.688064 28043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c98b1c3358949e5affc3e8acdad852a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40595 } }
I20260810 18:11:35.688544 28037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:35.689131 28043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c98b1c3358949e5affc3e8acdad852a. Latest consensus state: current_term: 1 leader_uuid: "7c98b1c3358949e5affc3e8acdad852a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c98b1c3358949e5affc3e8acdad852a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40595 } } }
I20260810 18:11:35.689254 28043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:35.689700 28045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c98b1c3358949e5affc3e8acdad852a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c98b1c3358949e5affc3e8acdad852a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40595 } } }
I20260810 18:11:35.689862 28045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:35.691461 28054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:35.694382 28054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:35.699743 28054 catalog_manager.cc:1347] Generated new cluster ID: 8c4651318d05457193e2df2cec4713ff
I20260810 18:11:35.699810 28054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:35.712378 28054 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:35.713552 28054 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:35.718164 28054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c98b1c3358949e5affc3e8acdad852a: Generated new TSK 0
I20260810 18:11:35.718907 28054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:11:35.748369 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40595
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:35.862100 28078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:35.862361 28078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:35.862402 28078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:35.866393 28078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:35.866451 28078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:35.866508 28078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:35.871727 28078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:35.872786 28078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:35.873785 28078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:35.883443 28086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:35.883460 28084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:35.883612 28078 server_base.cc:1034] running on GCE node
W20260810 18:11:35.883445 28088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:35.884020 28078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:35.884693 28078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:35.885890 28078 hybrid_clock.cc:648] HybridClock initialized: now 1786385495885872 us; error 33 us; skew 500 ppm
Aug 10 18:11:35 dist-test-slave-jpj1 krb5kdc[27905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385495, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:35.888687 28078 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:11:35.889694 28078 webserver.cc:460] Webserver started at http://127.23.65.65:33953/ using document root <none> and password file <none>
I20260810 18:11:35.890288 28078 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:35.890376 28078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:35.890589 28078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:35.892680 28078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "ae94e5d9a79a430a9cbd40d2d5ac5e7e"
format_stamp: "Formatted at 2026-08-10 18:11:35 on dist-test-slave-jpj1"
server_key: "b6b34e85c00101051dc789d92ac77073"
server_key_iv: "9c03fa0bd598fb2f20581d2b7d3dff0b"
server_key_version: "encryptionkey@0"
I20260810 18:11:35.893221 28078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "ae94e5d9a79a430a9cbd40d2d5ac5e7e"
format_stamp: "Formatted at 2026-08-10 18:11:35 on dist-test-slave-jpj1"
server_key: "b6b34e85c00101051dc789d92ac77073"
server_key_iv: "9c03fa0bd598fb2f20581d2b7d3dff0b"
server_key_version: "encryptionkey@0"
I20260810 18:11:35.896905 28078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:35.899389 28097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:35.900666 28078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:35.900774 28078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "ae94e5d9a79a430a9cbd40d2d5ac5e7e"
format_stamp: "Formatted at 2026-08-10 18:11:35 on dist-test-slave-jpj1"
server_key: "b6b34e85c00101051dc789d92ac77073"
server_key_iv: "9c03fa0bd598fb2f20581d2b7d3dff0b"
server_key_version: "encryptionkey@0"
I20260810 18:11:35.900873 28078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:35.911034 28078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:35.914144 28078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:35.914405 28078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:35.915040 28078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:35.915920 28078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:35.915972 28078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:35.916026 28078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:35.916055 28078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:35.924909 28078 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:39795
I20260810 18:11:35.924923 28282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:39795 every 8 connection(s)
I20260810 18:11:35.925859 28078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
Aug 10 18:11:35 dist-test-slave-jpj1 krb5kdc[27905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385495, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:35.935563 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 28078
I20260810 18:11:35.935667 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:35.935938 23813 external_mini_cluster.cc:1428] Setting key 9c9964afea2b2b2f37eda3f300ed5a59
I20260810 18:11:35.939515 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:35.927675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:42181 (local address 127.23.65.126:40595)
0810 18:11:35.927801 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:35.927805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:35.928887 (+ 1082us) server_negotiation.cc:407] Connection header received
0810 18:11:35.929805 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:35.929809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:35.929872 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:35.929962 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:35.931152 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:35.931736 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:35.933132 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:35.933385 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:35.933475 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:35.935977 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:35.935980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:35.935982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:35.936007 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:35.937538 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:35.938013 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:35.938017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:35.938019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:35.938096 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:35.938413 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:35.938417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:35.938419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:35.938593 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:35.938677 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:35.939222 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 18:11:35.939352 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:11:35.940326 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:35.928017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40595 (local address 127.23.65.65:42181)
0810 18:11:35.928725 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:35.928770 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:35.929536 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:35.930155 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:35.930163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:35.930530 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:35.931003 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:35.931017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:35.931859 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:35.931862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:35.932979 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:35.932991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:35.933525 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:35.933528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:35.933687 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:35.934251 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:35.934267 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:35.935815 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:35.937642 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:35.937655 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:35.937670 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:35.937921 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:35.938182 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:35.938185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:35.938187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:35.938327 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:35.938698 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:35.938704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:35.938945 (+ 241us) client_negotiation.cc:769] Sending connection context
0810 18:11:35.939153 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 18:11:35.939478 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":105,"threads_started":1}
I20260810 18:11:35.941607 28284 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40595
I20260810 18:11:35.941852 28284 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:35.942373 28284 heartbeater.cc:511] Master 127.23.65.126:40595 requested a full tablet report, sending...
I20260810 18:11:35.944141 27981 ts_manager.cc:194] Registered new tserver with Master: ae94e5d9a79a430a9cbd40d2d5ac5e7e (127.23.65.65:39795)
I20260810 18:11:35.945739 27981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:42181
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:11:35.966135 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40595
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:36.079748 28293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:36.080013 28293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:36.080070 28293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:36.084100 28293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:36.084162 28293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:36.084241 28293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:36.089632 28293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:36.090730 28293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:36.091812 28293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:36.101469 28301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:36.101449 28304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:36.101475 28302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:36.101786 28293 server_base.cc:1034] running on GCE node
I20260810 18:11:36.102236 28293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:36.102929 28293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:36.104127 28293 hybrid_clock.cc:648] HybridClock initialized: now 1786385496104098 us; error 46 us; skew 500 ppm
Aug 10 18:11:36 dist-test-slave-jpj1 krb5kdc[27905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:36.106940 28293 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:11:36.107931 28293 webserver.cc:460] Webserver started at http://127.23.65.66:36543/ using document root <none> and password file <none>
I20260810 18:11:36.108496 28293 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:36.108565 28293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:36.108769 28293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:36.110823 28293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "dd4e0d8a90034c24badffc7dbe95ad97"
format_stamp: "Formatted at 2026-08-10 18:11:36 on dist-test-slave-jpj1"
server_key: "40002345a567e2af145d5a847439e061"
server_key_iv: "d9871852d179e5bb49cdf6c17744d372"
server_key_version: "encryptionkey@0"
I20260810 18:11:36.111382 28293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "dd4e0d8a90034c24badffc7dbe95ad97"
format_stamp: "Formatted at 2026-08-10 18:11:36 on dist-test-slave-jpj1"
server_key: "40002345a567e2af145d5a847439e061"
server_key_iv: "d9871852d179e5bb49cdf6c17744d372"
server_key_version: "encryptionkey@0"
I20260810 18:11:36.115315 28293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 18:11:36.117865 28320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:36.119191 28293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:36.119309 28293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "dd4e0d8a90034c24badffc7dbe95ad97"
format_stamp: "Formatted at 2026-08-10 18:11:36 on dist-test-slave-jpj1"
server_key: "40002345a567e2af145d5a847439e061"
server_key_iv: "d9871852d179e5bb49cdf6c17744d372"
server_key_version: "encryptionkey@0"
I20260810 18:11:36.119416 28293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:36.142768 28293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:36.146044 28293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:36.146216 28293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:36.146793 28293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:36.147622 28293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:36.147678 28293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:36.147719 28293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:36.147738 28293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:36.156404 28293 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:40637
I20260810 18:11:36.156443 28513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:40637 every 8 connection(s)
I20260810 18:11:36.157321 28293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:36.163215 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 28293
I20260810 18:11:36.163312 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:36.163596 23813 external_mini_cluster.cc:1428] Setting key 6a2a096f8f4dc8853e7770ae5e13ca4b
Aug 10 18:11:36 dist-test-slave-jpj1 krb5kdc[27905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:36.170888 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:36.159129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:37975 (local address 127.23.65.126:40595)
0810 18:11:36.159299 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:36.159303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:36.160347 (+ 1044us) server_negotiation.cc:407] Connection header received
0810 18:11:36.161222 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:36.161226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:36.161282 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:36.161362 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:36.162536 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:36.163090 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:36.164353 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:36.164622 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:36.164715 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:36.167355 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:36.167358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:36.167360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:36.167385 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:36.168717 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:36.169200 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:36.169203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:36.169204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:36.169283 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:36.169765 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:36.169768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:36.169770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:36.169980 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:36.170079 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:36.170576 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 18:11:36.170717 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":62}
I20260810 18:11:36.171720 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:36.159513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40595 (local address 127.23.65.66:37975)
0810 18:11:36.160201 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:36.160243 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:36.161022 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:36.161544 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:36.161551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:36.161908 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:36.162394 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:36.162409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:36.163212 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:36.163214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:36.164221 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:36.164232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:36.164769 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:36.164773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:36.164954 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:36.165571 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:36.165589 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:36.167185 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:36.168831 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:36.168837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:36.168852 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:36.169104 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:36.169423 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:36.169428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:36.169429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:36.169658 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:36.170094 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:36.170101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:36.170342 (+ 241us) client_negotiation.cc:769] Sending connection context
0810 18:11:36.170511 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 18:11:36.170841 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":111,"threads_started":1}
I20260810 18:11:36.172890 28516 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40595
I20260810 18:11:36.173136 28516 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:11:36.173609 28516 heartbeater.cc:511] Master 127.23.65.126:40595 requested a full tablet report, sending...
I20260810 18:11:36.174608 27981 ts_manager.cc:194] Registered new tserver with Master: dd4e0d8a90034c24badffc7dbe95ad97 (127.23.65.66:40637)
I20260810 18:11:36.175180 27981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:37975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:11:36.192574 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40595
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:36.306838 28523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:36.307122 28523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:36.307188 28523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:36.311307 28523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:36.311375 28523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:36.311452 28523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:36.316673 28523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:36.317646 28523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:36.318622 28523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:36.328138 28533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:36.328192 28523 server_base.cc:1034] running on GCE node
W20260810 18:11:36.328138 28534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:36.328161 28537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:36.328744 28523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:36.329342 28523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:36.330497 28523 hybrid_clock.cc:648] HybridClock initialized: now 1786385496330476 us; error 36 us; skew 500 ppm
Aug 10 18:11:36 dist-test-slave-jpj1 krb5kdc[27905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:36.333119 28523 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:11:36.333954 28523 webserver.cc:460] Webserver started at http://127.23.65.67:40791/ using document root <none> and password file <none>
I20260810 18:11:36.334482 28523 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:36.334547 28523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:36.334733 28523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:36.336755 28523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "d589ce8254084a19a2307b5f879a03a2"
format_stamp: "Formatted at 2026-08-10 18:11:36 on dist-test-slave-jpj1"
server_key: "5674862bcdbc99889084a0e1beba4321"
server_key_iv: "f948cf6e6adc65c9245f819da95cc010"
server_key_version: "encryptionkey@0"
I20260810 18:11:36.337294 28523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "d589ce8254084a19a2307b5f879a03a2"
format_stamp: "Formatted at 2026-08-10 18:11:36 on dist-test-slave-jpj1"
server_key: "5674862bcdbc99889084a0e1beba4321"
server_key_iv: "f948cf6e6adc65c9245f819da95cc010"
server_key_version: "encryptionkey@0"
I20260810 18:11:36.340849 28523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:36.343128 28554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:36.344305 28523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:36.344434 28523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "d589ce8254084a19a2307b5f879a03a2"
format_stamp: "Formatted at 2026-08-10 18:11:36 on dist-test-slave-jpj1"
server_key: "5674862bcdbc99889084a0e1beba4321"
server_key_iv: "f948cf6e6adc65c9245f819da95cc010"
server_key_version: "encryptionkey@0"
I20260810 18:11:36.344522 28523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:36.390393 28523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:36.393278 28523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:36.393448 28523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:36.394132 28523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:36.394972 28523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:36.395030 28523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:36.395072 28523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:36.395092 28523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:36.404168 28523 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:45053
I20260810 18:11:36.404210 28744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:45053 every 8 connection(s)
I20260810 18:11:36.405133 28523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:36.409696 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 28523
I20260810 18:11:36.409813 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:36.410066 23813 external_mini_cluster.cc:1428] Setting key 7c5eac01e796b3a2baae8acb9490690b
Aug 10 18:11:36 dist-test-slave-jpj1 krb5kdc[27905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:36.418748 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:36.407002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:38669 (local address 127.23.65.126:40595)
0810 18:11:36.407148 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:36.407153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:36.408041 (+ 888us) server_negotiation.cc:407] Connection header received
0810 18:11:36.409329 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:36.409331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:36.409380 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:36.409480 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:36.410711 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:36.411413 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:36.412854 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:36.413079 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:36.413168 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:36.415494 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:36.415498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:36.415501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:36.415550 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:36.416743 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:36.417233 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:36.417237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:36.417238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:36.417325 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:36.417647 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:36.417650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:36.417652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:36.417851 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:36.417940 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:36.418461 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 18:11:36.418585 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:11:36.419580 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:36.407404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40595 (local address 127.23.65.67:38669)
0810 18:11:36.407890 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:36.407935 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:36.409128 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:36.409625 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:36.409632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:36.410018 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:36.410557 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:36.410576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:36.411577 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:36.411581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:36.412727 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:36.412738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:36.413220 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:36.413224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:36.413409 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:36.413921 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:36.413936 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:36.415332 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:36.416857 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:36.416864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:36.416879 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:36.417138 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:36.417410 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:36.417413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:36.417415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:36.417557 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:36.417954 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:36.417961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:36.418221 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 18:11:36.418396 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 18:11:36.418716 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":138,"threads_started":1}
I20260810 18:11:36.420596 28747 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40595
I20260810 18:11:36.420842 28747 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:36.421317 28747 heartbeater.cc:511] Master 127.23.65.126:40595 requested a full tablet report, sending...
I20260810 18:11:36.422459 27981 ts_manager.cc:194] Registered new tserver with Master: d589ce8254084a19a2307b5f879a03a2 (127.23.65.67:45053)
I20260810 18:11:36.423094 27981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:38669
I20260810 18:11:36.424142 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:11:36.439226 28041 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:36.437879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.23.65.126:40595)
0810 18:11:36.438025 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:36.438029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:36.438178 (+ 149us) server_negotiation.cc:407] Connection header received
0810 18:11:36.438290 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:36.438292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:36.438340 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:36.438411 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:36.439078 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47338: BlockingRecv error: recv got EOF from 127.0.0.1:47338 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:11:36.440927 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 28078
I20260810 18:11:36.472043 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 28293
I20260810 18:11:36.486825 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 28523
I20260810 18:11:36.499035 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 27930
2026-08-10T18:11:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2276 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:36 dist-test-slave-jpj1 krb5kdc[28769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:36 dist-test-slave-jpj1 krb5kdc[28769](info): set up 2 sockets
Aug 10 18:11:36 dist-test-slave-jpj1 krb5kdc[28769](info): commencing operation
krb5kdc: starting...
W20260810 18:11:37.663311 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:37 dist-test-slave-jpj1 krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:37.784421 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:42661
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:37.895725 28791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:37.895982 28791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:37.896019 28791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:37.900022 28791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:37.900077 28791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:37.900102 28791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:37.900133 28791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:37.900161 28791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:37.905334 28791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:42661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:42661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:37.906366 28791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:37.907370 28791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:11:37.915525 28791 server_base.cc:1034] running on GCE node
W20260810 18:11:37.915518 28801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:37.915478 28806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:37.915480 28803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:37.916492 28791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:37.917676 28791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:37.918841 28791 hybrid_clock.cc:648] HybridClock initialized: now 1786385497918817 us; error 40 us; skew 500 ppm
Aug 10 18:11:37 dist-test-slave-jpj1 krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:37.921399 28791 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:37.922215 28791 webserver.cc:460] Webserver started at http://127.23.65.126:38421/ using document root <none> and password file <none>
I20260810 18:11:37.922712 28791 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:37.922768 28791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:37.922942 28791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:37.924929 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "2d25091b96bd4acd9b42303f670e8f75"
format_stamp: "Formatted at 2026-08-10 18:11:37 on dist-test-slave-jpj1"
server_key: "c39f7af2a80c5606eb2eeb9b06908e86"
server_key_iv: "9631bdf2b33f4d31e9526dd54c13ffe1"
server_key_version: "encryptionkey@0"
I20260810 18:11:37.925439 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "2d25091b96bd4acd9b42303f670e8f75"
format_stamp: "Formatted at 2026-08-10 18:11:37 on dist-test-slave-jpj1"
server_key: "c39f7af2a80c5606eb2eeb9b06908e86"
server_key_iv: "9631bdf2b33f4d31e9526dd54c13ffe1"
server_key_version: "encryptionkey@0"
I20260810 18:11:37.929011 28791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260810 18:11:37.931341 28817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:37.932528 28791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:37.932636 28791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "2d25091b96bd4acd9b42303f670e8f75"
format_stamp: "Formatted at 2026-08-10 18:11:37 on dist-test-slave-jpj1"
server_key: "c39f7af2a80c5606eb2eeb9b06908e86"
server_key_iv: "9631bdf2b33f4d31e9526dd54c13ffe1"
server_key_version: "encryptionkey@0"
I20260810 18:11:37.932734 28791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:37.947732 28791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:37.950730 28791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:37.950912 28791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:37.957830 28791 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:42661
I20260810 18:11:37.958187 28896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:42661 every 8 connection(s)
I20260810 18:11:37.958811 28791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:37.960063 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 28791
I20260810 18:11:37.960178 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:37.960443 23813 external_mini_cluster.cc:1428] Setting key e9b550d882267c2cc104c1b12cbaa4ac
I20260810 18:11:37.962056 28897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:37.968565 28897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75: Bootstrap starting.
Aug 10 18:11:37 dist-test-slave-jpj1 krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:37.970963 28897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:37.971799 28897 log.cc:824] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:37.973615 28897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75: No bootstrap required, opened a new log
I20260810 18:11:37.974635 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:37.962024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.23.65.126:42661)
0810 18:11:37.962485 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:37.962500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:37.962537 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:11:37.963174 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:37.963190 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:37.963528 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:37.963855 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:37.964742 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:37.965726 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:37.966870 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:37.967341 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:37.967463 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:37.970236 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:37.970241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:37.970253 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:37.970281 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:37.971978 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:37.972394 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:37.972399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:37.972403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:37.972518 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:37.972780 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:37.972783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:37.972785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:37.973179 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:37.973369 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:37.973647 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 18:11:37.973926 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":146,"threads_started":1}
I20260810 18:11:37.977504 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:37.977573 28897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25091b96bd4acd9b42303f670e8f75" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 42661 } }
I20260810 18:11:37.977808 28897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:37.977865 28897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d25091b96bd4acd9b42303f670e8f75, State: Initialized, Role: FOLLOWER
I20260810 18:11:37.978256 28897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25091b96bd4acd9b42303f670e8f75" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 42661 } }
I20260810 18:11:37.978381 28897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:37.978444 28897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:37.978538 28897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:37.979389 28897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25091b96bd4acd9b42303f670e8f75" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 42661 } }
I20260810 18:11:37.979738 28897 leader_election.cc:302] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d25091b96bd4acd9b42303f670e8f75; no voters:
I20260810 18:11:37.980037 28897 leader_election.cc:288] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:37.980144 28908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:37.980329 28908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [term 1 LEADER]: Becoming Leader. State: Replica: 2d25091b96bd4acd9b42303f670e8f75, State: Running, Role: LEADER
I20260810 18:11:37.980629 28908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25091b96bd4acd9b42303f670e8f75" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 42661 } }
I20260810 18:11:37.981356 28897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:37.982221 28913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d25091b96bd4acd9b42303f670e8f75. Latest consensus state: current_term: 1 leader_uuid: "2d25091b96bd4acd9b42303f670e8f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25091b96bd4acd9b42303f670e8f75" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 42661 } } }
I20260810 18:11:37.982359 28913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:37.982697 28909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d25091b96bd4acd9b42303f670e8f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25091b96bd4acd9b42303f670e8f75" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 42661 } } }
I20260810 18:11:37.982806 28909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:37.983093 28923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:37.986370 28923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:37.992743 28923 catalog_manager.cc:1347] Generated new cluster ID: 4477c358a1ec4acea9e34e55c25a2ce1
I20260810 18:11:37.992810 28923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:37.999060 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:37.993469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.23.65.126:42661)
0810 18:11:37.993589 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:37.993594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:37.993956 (+ 362us) server_negotiation.cc:407] Connection header received
0810 18:11:37.994088 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:37.994091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:37.994135 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:37.994221 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:37.994857 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:37.995652 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:37.996809 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:37.997162 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:37.997243 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:37.998120 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:37.998124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:37.998127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:37.998631 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:37.998664 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:37.998683 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:37.998761 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:37.998861 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 18:11:37.998919 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41}
I20260810 18:11:38.004829 28923 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:38.005682 28923 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:38.011366 28923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d25091b96bd4acd9b42303f670e8f75: Generated new TSK 0
I20260810 18:11:38.012037 28923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:11:38.021921 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 28791
2026-08-10T18:11:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1526 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:38 dist-test-slave-jpj1 krb5kdc[28947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:38 dist-test-slave-jpj1 krb5kdc[28947](info): set up 2 sockets
Aug 10 18:11:38 dist-test-slave-jpj1 krb5kdc[28947](info): commencing operation
krb5kdc: starting...
W20260810 18:11:39.197476 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:39 dist-test-slave-jpj1 krb5kdc[28947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:39.324172 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46285
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:39.436337 28970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:39.436720 28970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:39.436785 28970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:39.440755 28970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:39.440806 28970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:39.440825 28970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:39.440841 28970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:39.440871 28970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:39.446043 28970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:39.447077 28970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:39.448153 28970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:39.456343 28983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:39.456300 28987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:39.456306 28984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:39.456575 28970 server_base.cc:1034] running on GCE node
I20260810 18:11:39.457243 28970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:39.458303 28970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:39.459471 28970 hybrid_clock.cc:648] HybridClock initialized: now 1786385499459451 us; error 32 us; skew 500 ppm
Aug 10 18:11:39 dist-test-slave-jpj1 krb5kdc[28947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385499, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:39.466459 28970 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:39.467351 28970 webserver.cc:460] Webserver started at http://127.23.65.126:35637/ using document root <none> and password file <none>
I20260810 18:11:39.467918 28970 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:39.467991 28970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:39.468204 28970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:39.470266 28970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "5561503757e14c65896862bccd053032"
format_stamp: "Formatted at 2026-08-10 18:11:39 on dist-test-slave-jpj1"
server_key: "5936c16a1a132047ae6c518f68e2e53b"
server_key_iv: "24773f94b33d6507f727056322dc835f"
server_key_version: "encryptionkey@0"
I20260810 18:11:39.470806 28970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "5561503757e14c65896862bccd053032"
format_stamp: "Formatted at 2026-08-10 18:11:39 on dist-test-slave-jpj1"
server_key: "5936c16a1a132047ae6c518f68e2e53b"
server_key_iv: "24773f94b33d6507f727056322dc835f"
server_key_version: "encryptionkey@0"
I20260810 18:11:39.474514 28970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:11:39.476927 28999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:39.478122 28970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:39.478241 28970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "5561503757e14c65896862bccd053032"
format_stamp: "Formatted at 2026-08-10 18:11:39 on dist-test-slave-jpj1"
server_key: "5936c16a1a132047ae6c518f68e2e53b"
server_key_iv: "24773f94b33d6507f727056322dc835f"
server_key_version: "encryptionkey@0"
I20260810 18:11:39.478345 28970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:39.492264 28970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:39.494969 28970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:39.495142 28970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:39.501952 28970 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:46285
I20260810 18:11:39.502197 29093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:46285 every 8 connection(s)
I20260810 18:11:39.502892 28970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:39.505604 29094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:39.510689 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 28970
I20260810 18:11:39.510800 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:39.510983 29094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032: Bootstrap starting.
I20260810 18:11:39.511050 23813 external_mini_cluster.cc:1428] Setting key 731ceb4030390a6d84467ba542c8cf11
I20260810 18:11:39.513444 29094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:39.514199 29094 log.cc:824] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:39.516491 29094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032: No bootstrap required, opened a new log
Aug 10 18:11:39 dist-test-slave-jpj1 krb5kdc[28947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:39.520685 29094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5561503757e14c65896862bccd053032" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46285 } }
I20260810 18:11:39.520905 29094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:39.520961 29094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5561503757e14c65896862bccd053032, State: Initialized, Role: FOLLOWER
I20260810 18:11:39.521435 29094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5561503757e14c65896862bccd053032" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46285 } }
I20260810 18:11:39.521579 29094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:39.521653 29094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:39.521757 29094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:39.522594 29094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5561503757e14c65896862bccd053032" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46285 } }
I20260810 18:11:39.522907 29094 leader_election.cc:302] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5561503757e14c65896862bccd053032; no voters:
I20260810 18:11:39.523207 29094 leader_election.cc:288] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:39.523303 29103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:39.523465 29103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [term 1 LEADER]: Becoming Leader. State: Replica: 5561503757e14c65896862bccd053032, State: Running, Role: LEADER
I20260810 18:11:39.523860 29103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5561503757e14c65896862bccd053032" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46285 } }
I20260810 18:11:39.524484 29094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:39.525094 29106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5561503757e14c65896862bccd053032. Latest consensus state: current_term: 1 leader_uuid: "5561503757e14c65896862bccd053032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5561503757e14c65896862bccd053032" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46285 } } }
I20260810 18:11:39.525208 29106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:39.525463 29105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5561503757e14c65896862bccd053032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5561503757e14c65896862bccd053032" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46285 } } }
I20260810 18:11:39.525554 29105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:39.525924 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:39.512350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.23.65.126:46285)
0810 18:11:39.512758 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:39.512774 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:39.512811 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:11:39.513440 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:39.513457 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:39.513803 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:39.514086 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:39.514955 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:39.516435 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:39.517561 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:39.518026 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:39.518147 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:39.520820 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:39.520826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:39.520839 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:39.520867 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:39.522488 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:39.522944 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:39.522950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:39.522956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:39.523098 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:39.523432 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:39.523436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:39.523438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:39.523913 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:39.524106 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:39.524673 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 18:11:39.524934 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":111,"threads_started":1}
W20260810 18:11:39.529183 29131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:39.529250 29131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:39.529543 29116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:39.530231 29116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:39.535398 29116 catalog_manager.cc:1347] Generated new cluster ID: 2f7b45f7eb9842ff910d63d05d07a169
I20260810 18:11:39.535472 29116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:39.549948 29116 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:39.551146 29116 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:39.560190 29116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5561503757e14c65896862bccd053032: Generated new TSK 0
I20260810 18:11:39.561010 29116 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:11:39.602473 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46285
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:39.742707 29138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:39.742952 29138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:39.742992 29138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:39.746860 29138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:39.746908 29138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:39.746963 29138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:39.752099 29138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:39.753049 29138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:39.754196 29138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:39.763785 29147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:39.763936 29138 server_base.cc:1034] running on GCE node
W20260810 18:11:39.763758 29145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:39.763767 29144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:39.764459 29138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:39.765086 29138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:39.766247 29138 hybrid_clock.cc:648] HybridClock initialized: now 1786385499766229 us; error 28 us; skew 500 ppm
Aug 10 18:11:39 dist-test-slave-jpj1 krb5kdc[28947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385499, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:39.768911 29138 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:11:39.769786 29138 webserver.cc:460] Webserver started at http://127.23.65.65:37201/ using document root <none> and password file <none>
I20260810 18:11:39.770351 29138 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:39.770418 29138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:39.770607 29138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:39.772888 29138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "00576d1881f947cf81a70eeccd451478"
format_stamp: "Formatted at 2026-08-10 18:11:39 on dist-test-slave-jpj1"
server_key: "20331a6e2b41ade135282d9137800d0d"
server_key_iv: "03ac93a0d4f6c94b404e77c728ad768d"
server_key_version: "encryptionkey@0"
I20260810 18:11:39.773561 29138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "00576d1881f947cf81a70eeccd451478"
format_stamp: "Formatted at 2026-08-10 18:11:39 on dist-test-slave-jpj1"
server_key: "20331a6e2b41ade135282d9137800d0d"
server_key_iv: "03ac93a0d4f6c94b404e77c728ad768d"
server_key_version: "encryptionkey@0"
I20260810 18:11:39.777879 29138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:11:39.780227 29162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:39.781428 29138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:11:39.781535 29138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "00576d1881f947cf81a70eeccd451478"
format_stamp: "Formatted at 2026-08-10 18:11:39 on dist-test-slave-jpj1"
server_key: "20331a6e2b41ade135282d9137800d0d"
server_key_iv: "03ac93a0d4f6c94b404e77c728ad768d"
server_key_version: "encryptionkey@0"
I20260810 18:11:39.781618 29138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:39.818338 29138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:39.821404 29138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:39.821565 29138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:39.822130 29138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:39.822949 29138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:39.822999 29138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:39.823040 29138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:39.823053 29138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:39.831785 29138 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:38373
I20260810 18:11:39.831815 29362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:38373 every 8 connection(s)
I20260810 18:11:39.832664 29138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:39.839635 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 29138
I20260810 18:11:39.839757 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:39.840006 23813 external_mini_cluster.cc:1428] Setting key 0a193044016b87cb1f0207bb1daa2727
Aug 10 18:11:39 dist-test-slave-jpj1 krb5kdc[28947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385499, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:39.846122 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:39.834402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:50319 (local address 127.23.65.126:46285)
0810 18:11:39.834522 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:39.834525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:39.835363 (+ 838us) server_negotiation.cc:407] Connection header received
0810 18:11:39.836797 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:39.836800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:39.836857 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:39.836951 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:39.838265 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:39.838811 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:39.840050 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:39.840320 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:39.840415 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:39.842867 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:39.842870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:39.842872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:39.842896 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:39.844228 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:39.844697 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:39.844701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:39.844703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:39.844781 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:39.845086 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:39.845089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:39.845091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:39.845272 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:39.845357 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:39.845836 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 18:11:39.845959 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:11:39.846899 29367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:39.834736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46285 (local address 127.23.65.65:50319)
0810 18:11:39.835203 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:39.835245 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:39.836602 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:39.837155 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:39.837162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:39.837609 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:39.838111 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:39.838125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:39.838902 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:39.838905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:39.839928 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:39.839938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:39.840469 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:39.840473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:39.840660 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:39.841260 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:39.841280 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:39.842713 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:39.844336 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:39.844342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:39.844357 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:39.844602 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:39.844862 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:39.844865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:39.844867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:39.844999 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:39.845372 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:39.845379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:39.845608 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 18:11:39.845776 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 18:11:39.846084 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
I20260810 18:11:39.848069 29363 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46285
I20260810 18:11:39.848351 29363 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:39.848944 29363 heartbeater.cc:511] Master 127.23.65.126:46285 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:11:39.850669 29025 ts_manager.cc:194] Registered new tserver with Master: 00576d1881f947cf81a70eeccd451478 (127.23.65.65:38373)
I20260810 18:11:39.851903 29025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:50319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:11:39.868429 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46285
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:39.978892 29370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:39.979147 29370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:39.979204 29370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:39.983170 29370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:39.983223 29370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:39.983299 29370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:39.988461 29370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:39.989434 29370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:39.990427 29370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:39.999606 29384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:39.999562 29377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:39.999559 29378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:39.999660 29370 server_base.cc:1034] running on GCE node
I20260810 18:11:40.000180 29370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:40.000779 29370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:40.001941 29370 hybrid_clock.cc:648] HybridClock initialized: now 1786385500001924 us; error 32 us; skew 500 ppm
Aug 10 18:11:40 dist-test-slave-jpj1 krb5kdc[28947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385500, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:40.004624 29370 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:11:40.005502 29370 webserver.cc:460] Webserver started at http://127.23.65.66:37845/ using document root <none> and password file <none>
I20260810 18:11:40.006042 29370 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:40.006110 29370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:40.006302 29370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:40.008399 29370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "81684731528040ce8fa2cbfe1d60bf65"
format_stamp: "Formatted at 2026-08-10 18:11:40 on dist-test-slave-jpj1"
server_key: "926c2725bb83973f279ac68b56aacd06"
server_key_iv: "c7fb10498a3c0f766132bfe820909851"
server_key_version: "encryptionkey@0"
I20260810 18:11:40.008951 29370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "81684731528040ce8fa2cbfe1d60bf65"
format_stamp: "Formatted at 2026-08-10 18:11:40 on dist-test-slave-jpj1"
server_key: "926c2725bb83973f279ac68b56aacd06"
server_key_iv: "c7fb10498a3c0f766132bfe820909851"
server_key_version: "encryptionkey@0"
I20260810 18:11:40.012609 29370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:11:40.014998 29393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:40.016273 29370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:40.016412 29370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "81684731528040ce8fa2cbfe1d60bf65"
format_stamp: "Formatted at 2026-08-10 18:11:40 on dist-test-slave-jpj1"
server_key: "926c2725bb83973f279ac68b56aacd06"
server_key_iv: "c7fb10498a3c0f766132bfe820909851"
server_key_version: "encryptionkey@0"
I20260810 18:11:40.016508 29370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:40.050091 29370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:40.052980 29370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:40.053170 29370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:40.053783 29370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:40.054634 29370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:40.054685 29370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:40.054741 29370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:40.054770 29370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:40.063400 29370 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:45087
I20260810 18:11:40.063436 29590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:45087 every 8 connection(s)
I20260810 18:11:40.064304 29370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:40.065316 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 29370
I20260810 18:11:40.065423 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:40.065712 23813 external_mini_cluster.cc:1428] Setting key b8460d0f91a9bd150db0eca17c80e72c
Aug 10 18:11:40 dist-test-slave-jpj1 krb5kdc[28947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385500, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
I20260810 18:11:40.078580 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:40.066292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:43957 (local address 127.23.65.126:46285)
0810 18:11:40.066428 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:40.066432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:40.067499 (+ 1067us) server_negotiation.cc:407] Connection header received
0810 18:11:40.069236 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:40.069239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:40.069284 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:40.069364 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:40.070544 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.071146 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.072320 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.072631 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.072726 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:40.075194 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:40.075198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:40.075201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:40.075225 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:40.076638 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:40.077122 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:40.077124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:40.077126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:40.077204 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:40.077481 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:40.077483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:40.077485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:40.077680 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:40.077802 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:40.078306 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 18:11:40.078419 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:11:40.079595 29596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:40.066769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46285 (local address 127.23.65.66:43957)
0810 18:11:40.067334 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:40.067394 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:40.069022 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:40.069553 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:40.069562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:40.069927 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:40.070405 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:40.070419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.071280 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.071283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:40.072199 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:40.072210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.072785 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.072790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:40.072965 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:40.073579 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:40.073596 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:40.075031 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:40.076742 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:40.076747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:40.076760 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:40.077027 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:40.077270 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:40.077272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:40.077273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:40.077405 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:40.077769 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:40.077774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:40.078064 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 18:11:40.078272 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 18:11:40.078614 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":136,"threads_started":1}
I20260810 18:11:40.080840 29591 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46285
I20260810 18:11:40.081097 29591 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:40.081631 29591 heartbeater.cc:511] Master 127.23.65.126:46285 requested a full tablet report, sending...
I20260810 18:11:40.082707 29025 ts_manager.cc:194] Registered new tserver with Master: 81684731528040ce8fa2cbfe1d60bf65 (127.23.65.66:45087)
I20260810 18:11:40.083316 29025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:43957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:11:40.094691 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46285
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:40.205982 29601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:40.206218 29601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:40.206262 29601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:40.210366 29601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:40.210436 29601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:40.210506 29601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:40.215963 29601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:40.216974 29601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:40.217990 29601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:40.227834 29608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:40.227836 29607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:40.227881 29611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:40.228008 29601 server_base.cc:1034] running on GCE node
I20260810 18:11:40.228490 29601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:40.229066 29601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:40.230226 29601 hybrid_clock.cc:648] HybridClock initialized: now 1786385500230198 us; error 32 us; skew 500 ppm
Aug 10 18:11:40 dist-test-slave-jpj1 krb5kdc[28947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385500, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:40.232868 29601 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:11:40.233701 29601 webserver.cc:460] Webserver started at http://127.23.65.67:43459/ using document root <none> and password file <none>
I20260810 18:11:40.234215 29601 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:40.234288 29601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:40.234486 29601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:40.236514 29601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "d16296b5185146889ad560622228ca8b"
format_stamp: "Formatted at 2026-08-10 18:11:40 on dist-test-slave-jpj1"
server_key: "0fe42b3b04923b996a359b9f3cbcde4b"
server_key_iv: "bb5c2d2922c20535aeec0dd066118317"
server_key_version: "encryptionkey@0"
I20260810 18:11:40.237042 29601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "d16296b5185146889ad560622228ca8b"
format_stamp: "Formatted at 2026-08-10 18:11:40 on dist-test-slave-jpj1"
server_key: "0fe42b3b04923b996a359b9f3cbcde4b"
server_key_iv: "bb5c2d2922c20535aeec0dd066118317"
server_key_version: "encryptionkey@0"
I20260810 18:11:40.240746 29601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:40.243040 29621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:40.244249 29601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:40.244366 29601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "d16296b5185146889ad560622228ca8b"
format_stamp: "Formatted at 2026-08-10 18:11:40 on dist-test-slave-jpj1"
server_key: "0fe42b3b04923b996a359b9f3cbcde4b"
server_key_iv: "bb5c2d2922c20535aeec0dd066118317"
server_key_version: "encryptionkey@0"
I20260810 18:11:40.244462 29601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:40.305537 29601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:40.308718 29601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:40.308907 29601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:40.309520 29601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:40.310365 29601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:40.310415 29601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:40.310472 29601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:40.310497 29601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:40.318884 29601 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:42241
I20260810 18:11:40.318923 29805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:42241 every 8 connection(s)
I20260810 18:11:40.319873 29601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:40.321827 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 29601
I20260810 18:11:40.321941 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:40.322193 23813 external_mini_cluster.cc:1428] Setting key 25ce01112eb811b3401fb1b51696f461
Aug 10 18:11:40 dist-test-slave-jpj1 krb5kdc[28947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385500, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:40.332594 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:40.321635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:34619 (local address 127.23.65.126:46285)
0810 18:11:40.321789 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:40.321794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:40.322646 (+ 852us) server_negotiation.cc:407] Connection header received
0810 18:11:40.324019 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:40.324022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:40.324073 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:40.324159 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:40.325381 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.325944 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.327059 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.327296 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.327374 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:40.329587 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:40.329590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:40.329592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:40.329618 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:40.330866 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:40.331308 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:40.331310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:40.331311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:40.331381 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:40.331658 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:40.331661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:40.331662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:40.331826 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:40.331897 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:40.332378 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 18:11:40.332469 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:11:40.333379 29808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:40.322018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46285 (local address 127.23.65.67:34619)
0810 18:11:40.322501 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:40.322555 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:40.323836 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:40.324326 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:40.324333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:40.324687 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:40.325230 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:40.325246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.326051 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.326053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:40.326954 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:40.326964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.327407 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.327410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:40.327564 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:40.328046 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:40.328061 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:40.329430 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:40.330965 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:40.330970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:40.330980 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:40.331222 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:40.331442 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:40.331444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:40.331445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:40.331592 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:40.331940 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:40.331946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:40.332182 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 18:11:40.332336 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 18:11:40.332573 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260810 18:11:40.334471 29806 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46285
I20260810 18:11:40.334714 29806 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:40.335254 29806 heartbeater.cc:511] Master 127.23.65.126:46285 requested a full tablet report, sending...
I20260810 18:11:40.336279 29025 ts_manager.cc:194] Registered new tserver with Master: d16296b5185146889ad560622228ca8b (127.23.65.67:42241)
I20260810 18:11:40.336813 29025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:34619
I20260810 18:11:40.347118 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:11:40 dist-test-slave-jpj1 krb5kdc[28947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385500, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 18:11:40 dist-test-slave-jpj1 krb5kdc[28947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385500, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:40.372846 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:40.365395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.23.65.126:46285)
0810 18:11:40.365512 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:40.365516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:40.365640 (+ 124us) server_negotiation.cc:407] Connection header received
0810 18:11:40.365743 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:40.365746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:40.365790 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:40.365861 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:40.366498 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.366998 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.367791 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:40.367995 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:40.368068 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:40.370175 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:40.370179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:40.370181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:40.370205 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:40.371506 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:40.371876 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:40.371879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:40.371880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:40.371956 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:40.372148 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:40.372150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:40.372151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:40.372377 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:40.372478 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:40.372630 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 18:11:40.372723 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 18:11:40.373241 29025 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37776
I20260810 18:11:40.374051 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 29138
I20260810 18:11:40.394690 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 29370
I20260810 18:11:40.404451 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 29601
I20260810 18:11:40.419795 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 28970
2026-08-10T18:11:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2395 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:40 dist-test-slave-jpj1 krb5kdc[29823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:40 dist-test-slave-jpj1 krb5kdc[29823](info): set up 2 sockets
Aug 10 18:11:40 dist-test-slave-jpj1 krb5kdc[29823](info): commencing operation
krb5kdc: starting...
W20260810 18:11:41.579311 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:41 dist-test-slave-jpj1 krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:41.698611 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46581
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:41.808457 29842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:41.808713 29842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:41.808750 29842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:41.812644 29842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:41.812701 29842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:41.812719 29842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:41.812734 29842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:41.812748 29842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:41.817875 29842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:41.818857 29842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:41.819856 29842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:41.827982 29848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:41.827993 29849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:41.828001 29856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:41.828145 29842 server_base.cc:1034] running on GCE node
I20260810 18:11:41.828826 29842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:41.829876 29842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:41.831043 29842 hybrid_clock.cc:648] HybridClock initialized: now 1786385501831027 us; error 27 us; skew 500 ppm
Aug 10 18:11:41 dist-test-slave-jpj1 krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385501, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:41.833668 29842 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:41.834481 29842 webserver.cc:460] Webserver started at http://127.23.65.126:32843/ using document root <none> and password file <none>
I20260810 18:11:41.835000 29842 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:41.835067 29842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:41.835258 29842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:41.837231 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "d09dccb54c4f49b38f640b202dab3a55"
format_stamp: "Formatted at 2026-08-10 18:11:41 on dist-test-slave-jpj1"
server_key: "ccbde77b96ce138b3ea9166f6069dbca"
server_key_iv: "66f0cd94eeb322c87bc4ba961b34022d"
server_key_version: "encryptionkey@0"
I20260810 18:11:41.837778 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "d09dccb54c4f49b38f640b202dab3a55"
format_stamp: "Formatted at 2026-08-10 18:11:41 on dist-test-slave-jpj1"
server_key: "ccbde77b96ce138b3ea9166f6069dbca"
server_key_iv: "66f0cd94eeb322c87bc4ba961b34022d"
server_key_version: "encryptionkey@0"
I20260810 18:11:41.841331 29842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:11:41.843523 29868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:41.844705 29842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:11:41.844817 29842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "d09dccb54c4f49b38f640b202dab3a55"
format_stamp: "Formatted at 2026-08-10 18:11:41 on dist-test-slave-jpj1"
server_key: "ccbde77b96ce138b3ea9166f6069dbca"
server_key_iv: "66f0cd94eeb322c87bc4ba961b34022d"
server_key_version: "encryptionkey@0"
I20260810 18:11:41.844903 29842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:41.867589 29842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:41.870347 29842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:41.870522 29842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:41.877223 29842 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:46581
I20260810 18:11:41.877292 29962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:46581 every 8 connection(s)
I20260810 18:11:41.878140 29842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:41.880852 29965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:41.884591 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 29842
I20260810 18:11:41.884702 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:41.884960 23813 external_mini_cluster.cc:1428] Setting key e697cd51bce439a114833c454a43f1e0
I20260810 18:11:41.886783 29965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55: Bootstrap starting.
I20260810 18:11:41.888962 29965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:41.889688 29965 log.cc:824] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:41.891482 29965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55: No bootstrap required, opened a new log
Aug 10 18:11:41 dist-test-slave-jpj1 krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:41.895418 29965 raft_consensus.cc:348] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09dccb54c4f49b38f640b202dab3a55" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46581 } }
I20260810 18:11:41.895668 29965 raft_consensus.cc:374] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:41.895728 29965 raft_consensus.cc:729] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09dccb54c4f49b38f640b202dab3a55, State: Initialized, Role: FOLLOWER
I20260810 18:11:41.896158 29965 consensus_queue.cc:260] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09dccb54c4f49b38f640b202dab3a55" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46581 } }
I20260810 18:11:41.896306 29965 raft_consensus.cc:388] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:41.896373 29965 raft_consensus.cc:482] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:41.896469 29965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:41.897323 29965 raft_consensus.cc:504] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09dccb54c4f49b38f640b202dab3a55" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46581 } }
I20260810 18:11:41.897665 29965 leader_election.cc:302] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d09dccb54c4f49b38f640b202dab3a55; no voters:
I20260810 18:11:41.897974 29965 leader_election.cc:288] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:41.898054 29971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:41.898221 29971 raft_consensus.cc:686] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [term 1 LEADER]: Becoming Leader. State: Replica: d09dccb54c4f49b38f640b202dab3a55, State: Running, Role: LEADER
I20260810 18:11:41.898545 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:41.886288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.23.65.126:46581)
0810 18:11:41.886713 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:41.886729 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:41.886784 (+ 55us) server_negotiation.cc:407] Connection header received
0810 18:11:41.887432 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:41.887450 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:41.887817 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:41.888117 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:41.888935 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:41.890003 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:41.890822 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:41.891275 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:41.891419 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:41.893942 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:41.893947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:41.893960 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:41.893986 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:41.895526 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:41.896014 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:41.896019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:41.896023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:41.896135 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:41.896418 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:41.896421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:41.896423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:41.896851 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:41.897043 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:41.897317 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 18:11:41.897627 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260810 18:11:41.898573 29971 consensus_queue.cc:237] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09dccb54c4f49b38f640b202dab3a55" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46581 } }
I20260810 18:11:41.899225 29965 sys_catalog.cc:564] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:41.901036 29974 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d09dccb54c4f49b38f640b202dab3a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09dccb54c4f49b38f640b202dab3a55" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46581 } } }
I20260810 18:11:41.901302 29974 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:41.901332 29973 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d09dccb54c4f49b38f640b202dab3a55. Latest consensus state: current_term: 1 leader_uuid: "d09dccb54c4f49b38f640b202dab3a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09dccb54c4f49b38f640b202dab3a55" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46581 } } }
I20260810 18:11:41.901573 29973 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55 [sys.catalog]: This master's current role is: LEADER
W20260810 18:11:41.903837 29996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:41.903903 29996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:41.903959 29998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:41.904608 29998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:41.909772 29998 catalog_manager.cc:1347] Generated new cluster ID: 627be8b075594708b0c77c5e0c190de6
I20260810 18:11:41.909832 29998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:41.926558 29998 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:41.927924 29998 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:41.944181 29998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d09dccb54c4f49b38f640b202dab3a55: Generated new TSK 0
I20260810 18:11:41.944943 29998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:11:41.982219 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46581
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:42.095781 30008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:42.096040 30008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:42.096086 30008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:42.100214 30008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:42.100273 30008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:42.100339 30008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:42.105641 30008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:42.106688 30008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:42.107756 30008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:42.117197 30015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:42.117250 30016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:42.117218 30020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:42.117624 30008 server_base.cc:1034] running on GCE node
I20260810 18:11:42.118052 30008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:42.118664 30008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:42.119853 30008 hybrid_clock.cc:648] HybridClock initialized: now 1786385502119825 us; error 39 us; skew 500 ppm
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385502, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:42.122758 30008 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:11:42.123629 30008 webserver.cc:460] Webserver started at http://127.23.65.65:34963/ using document root <none> and password file <none>
I20260810 18:11:42.124151 30008 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:42.124210 30008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:42.124389 30008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:42.126372 30008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "d84cd1b888684a3aab014b1258c96775"
format_stamp: "Formatted at 2026-08-10 18:11:42 on dist-test-slave-jpj1"
server_key: "d94b93b7a6d5485d354cc8bb4d1f5c3d"
server_key_iv: "fe77c30ba15c56aa35430eefa3180a74"
server_key_version: "encryptionkey@0"
I20260810 18:11:42.126889 30008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "d84cd1b888684a3aab014b1258c96775"
format_stamp: "Formatted at 2026-08-10 18:11:42 on dist-test-slave-jpj1"
server_key: "d94b93b7a6d5485d354cc8bb4d1f5c3d"
server_key_iv: "fe77c30ba15c56aa35430eefa3180a74"
server_key_version: "encryptionkey@0"
I20260810 18:11:42.130528 30008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:11:42.132936 30027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:42.134096 30008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:42.134197 30008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "d84cd1b888684a3aab014b1258c96775"
format_stamp: "Formatted at 2026-08-10 18:11:42 on dist-test-slave-jpj1"
server_key: "d94b93b7a6d5485d354cc8bb4d1f5c3d"
server_key_iv: "fe77c30ba15c56aa35430eefa3180a74"
server_key_version: "encryptionkey@0"
I20260810 18:11:42.134296 30008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:42.151062 30008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:42.153910 30008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:42.154110 30008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:42.154721 30008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:42.155596 30008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:42.155642 30008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:42.155702 30008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:42.155730 30008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:42.164425 30008 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:37357
I20260810 18:11:42.164458 30213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:37357 every 8 connection(s)
I20260810 18:11:42.165324 30008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:42.168804 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 30008
I20260810 18:11:42.168921 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:42.169186 23813 external_mini_cluster.cc:1428] Setting key f361b99d8cff62771f66e29167357617
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385502, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:11:42.179785 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:42.167125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:41349 (local address 127.23.65.126:46581)
0810 18:11:42.167267 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:42.167271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:42.168144 (+ 873us) server_negotiation.cc:407] Connection header received
0810 18:11:42.169590 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:42.169593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:42.169651 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:42.169728 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:42.170908 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.171437 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.172784 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.173022 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.173110 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:42.175994 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:42.175997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:42.175999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:42.176025 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:42.177409 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.178059 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.178062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.178064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.178148 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.178577 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.178579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.178580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.178775 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:42.178846 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:42.179498 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 18:11:42.179651 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:11:42.180708 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:42.167483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46581 (local address 127.23.65.65:41349)
0810 18:11:42.167996 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:42.168041 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:42.169371 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:42.169919 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:42.169926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:42.170278 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:42.170759 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:42.170773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.171579 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.171583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:42.172648 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:42.172662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.173171 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.173175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:42.173370 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:42.173990 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:42.174007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:42.175820 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:42.177530 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:42.177537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:42.177553 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:42.177928 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:42.178257 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:42.178261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:42.178263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:42.178482 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:42.178884 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:42.178891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:42.179230 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 18:11:42.179453 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 18:11:42.179841 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":116,"threads_started":1}
I20260810 18:11:42.181907 30214 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
I20260810 18:11:42.182180 30214 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:42.182713 30214 heartbeater.cc:511] Master 127.23.65.126:46581 requested a full tablet report, sending...
I20260810 18:11:42.184297 29901 ts_manager.cc:194] Registered new tserver with Master: d84cd1b888684a3aab014b1258c96775 (127.23.65.65:37357)
I20260810 18:11:42.185665 29901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:41349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:11:42.198127 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46581
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:42.309962 30223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:42.310204 30223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:42.310261 30223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:42.314222 30223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:42.314275 30223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:42.314352 30223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:42.319365 30223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:42.320369 30223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:42.321342 30223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:42.330947 30229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:42.331004 30223 server_base.cc:1034] running on GCE node
W20260810 18:11:42.330935 30228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:42.330947 30234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:42.331573 30223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:42.332170 30223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:42.333328 30223 hybrid_clock.cc:648] HybridClock initialized: now 1786385502333308 us; error 32 us; skew 500 ppm
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385502, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:42.336011 30223 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:11:42.336858 30223 webserver.cc:460] Webserver started at http://127.23.65.66:35927/ using document root <none> and password file <none>
I20260810 18:11:42.337381 30223 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:42.337443 30223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:42.337637 30223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:42.339643 30223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "1671ad4c454e4a96910f268f5adb9736"
format_stamp: "Formatted at 2026-08-10 18:11:42 on dist-test-slave-jpj1"
server_key: "f8f30f77511947b09286f7b3500e1e45"
server_key_iv: "524924b726a805ce47d4e8ef37942f30"
server_key_version: "encryptionkey@0"
I20260810 18:11:42.340224 30223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "1671ad4c454e4a96910f268f5adb9736"
format_stamp: "Formatted at 2026-08-10 18:11:42 on dist-test-slave-jpj1"
server_key: "f8f30f77511947b09286f7b3500e1e45"
server_key_iv: "524924b726a805ce47d4e8ef37942f30"
server_key_version: "encryptionkey@0"
I20260810 18:11:42.343838 30223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:11:42.346081 30248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:42.347220 30223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:42.347339 30223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "1671ad4c454e4a96910f268f5adb9736"
format_stamp: "Formatted at 2026-08-10 18:11:42 on dist-test-slave-jpj1"
server_key: "f8f30f77511947b09286f7b3500e1e45"
server_key_iv: "524924b726a805ce47d4e8ef37942f30"
server_key_version: "encryptionkey@0"
I20260810 18:11:42.347430 30223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:42.360075 30223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:42.362612 30223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:42.362779 30223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:42.363338 30223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:42.364164 30223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:42.364212 30223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:42.364264 30223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:42.364290 30223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:42.373080 30223 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:42483
I20260810 18:11:42.373127 30438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:42483 every 8 connection(s)
I20260810 18:11:42.373965 30223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:42.374482 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 30223
I20260810 18:11:42.374595 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:42.374894 23813 external_mini_cluster.cc:1428] Setting key d2d9255d7b336d9ab8acdd997a24346f
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385502, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
I20260810 18:11:42.387341 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:42.375828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:36541 (local address 127.23.65.126:46581)
0810 18:11:42.375966 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:42.375970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:42.376958 (+ 988us) server_negotiation.cc:407] Connection header received
0810 18:11:42.378305 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:42.378309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:42.378365 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:42.378439 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:42.379626 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.380158 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.381276 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.381507 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.381596 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:42.383969 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:42.383973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:42.383976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:42.384000 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:42.385282 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.385783 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.385786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.385788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.385876 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.386196 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.386199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.386201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.386394 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:42.386522 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:42.387051 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 18:11:42.387179 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:11:42.388149 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:42.376299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46581 (local address 127.23.65.66:36541)
0810 18:11:42.376812 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:42.376859 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:42.378104 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:42.378615 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:42.378622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:42.378971 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:42.379452 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:42.379466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.380282 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.380285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:42.381155 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:42.381165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.381650 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.381654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:42.381809 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:42.382400 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:42.382416 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:42.383810 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:42.385395 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:42.385401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:42.385415 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:42.385689 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:42.385962 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:42.385965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:42.385967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:42.386106 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:42.386501 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:42.386508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:42.386769 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 18:11:42.386937 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 18:11:42.387247 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":148,"threads_started":1}
I20260810 18:11:42.389225 30442 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46581
I20260810 18:11:42.389495 30442 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:42.390084 30442 heartbeater.cc:511] Master 127.23.65.126:46581 requested a full tablet report, sending...
I20260810 18:11:42.391186 29901 ts_manager.cc:194] Registered new tserver with Master: 1671ad4c454e4a96910f268f5adb9736 (127.23.65.66:42483)
I20260810 18:11:42.391770 29901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:36541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:11:42.402105 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46581
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:42.514402 30455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:42.514652 30455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:42.514710 30455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:42.518743 30455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:42.518808 30455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:42.518887 30455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:42.524204 30455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:42.525213 30455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:42.526201 30455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:42.535704 30466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:42.535710 30464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:42.535746 30455 server_base.cc:1034] running on GCE node
W20260810 18:11:42.535710 30462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:42.536417 30455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:42.537047 30455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:42.538201 30455 hybrid_clock.cc:648] HybridClock initialized: now 1786385502538172 us; error 40 us; skew 500 ppm
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385502, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:42.540925 30455 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:11:42.541743 30455 webserver.cc:460] Webserver started at http://127.23.65.67:37663/ using document root <none> and password file <none>
I20260810 18:11:42.542335 30455 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:42.542400 30455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:42.542569 30455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:42.544634 30455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "e9f121978f3b4329bce8afe562ac432e"
format_stamp: "Formatted at 2026-08-10 18:11:42 on dist-test-slave-jpj1"
server_key: "00c1ecb6f7e21ffe9101f4adeb439575"
server_key_iv: "ac6946100a3568598da425018e49613b"
server_key_version: "encryptionkey@0"
I20260810 18:11:42.545150 30455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "e9f121978f3b4329bce8afe562ac432e"
format_stamp: "Formatted at 2026-08-10 18:11:42 on dist-test-slave-jpj1"
server_key: "00c1ecb6f7e21ffe9101f4adeb439575"
server_key_iv: "ac6946100a3568598da425018e49613b"
server_key_version: "encryptionkey@0"
I20260810 18:11:42.548872 30455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:42.551256 30476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:42.552518 30455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:42.552627 30455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "e9f121978f3b4329bce8afe562ac432e"
format_stamp: "Formatted at 2026-08-10 18:11:42 on dist-test-slave-jpj1"
server_key: "00c1ecb6f7e21ffe9101f4adeb439575"
server_key_iv: "ac6946100a3568598da425018e49613b"
server_key_version: "encryptionkey@0"
I20260810 18:11:42.552711 30455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:42.569841 30455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:42.572467 30455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:42.572615 30455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:42.573146 30455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:42.573923 30455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:42.573971 30455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:42.574009 30455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:42.574029 30455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:42.582335 30455 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:37385
I20260810 18:11:42.582357 30686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:37385 every 8 connection(s)
I20260810 18:11:42.583176 30455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:42.588022 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 30455
I20260810 18:11:42.588145 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:42.588397 23813 external_mini_cluster.cc:1428] Setting key 2aebc69cddc835d4bb2bde87c169bf5f
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385502, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:42.596112 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:42.584897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:40125 (local address 127.23.65.126:46581)
0810 18:11:42.585038 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:42.585042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:42.585835 (+ 793us) server_negotiation.cc:407] Connection header received
0810 18:11:42.587164 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:42.587167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:42.587223 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:42.587295 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:42.588540 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.589197 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.590428 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.590646 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.590734 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:42.593040 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:42.593043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:42.593045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:42.593069 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:42.594157 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.594631 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.594634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.594635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.594713 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.595024 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.595028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.595030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.595205 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:42.595289 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:42.595827 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 18:11:42.595953 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:11:42.596885 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:42.585236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46581 (local address 127.23.65.67:40125)
0810 18:11:42.585690 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:42.585734 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:42.586957 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:42.587484 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:42.587490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:42.587886 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:42.588398 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:42.588414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.589331 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.589334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:42.590289 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:42.590301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.590800 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.590805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:42.590972 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:42.591471 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:42.591487 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:42.592883 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:42.594264 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:42.594271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:42.594284 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:42.594534 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:42.594797 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:42.594800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:42.594802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:42.594933 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:42.595308 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:42.595326 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:42.595591 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 18:11:42.595760 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 18:11:42.596064 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260810 18:11:42.597867 30687 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46581
I20260810 18:11:42.598078 30687 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:42.598524 30687 heartbeater.cc:511] Master 127.23.65.126:46581 requested a full tablet report, sending...
I20260810 18:11:42.599454 29901 ts_manager.cc:194] Registered new tserver with Master: e9f121978f3b4329bce8afe562ac432e (127.23.65.67:37385)
I20260810 18:11:42.600023 29901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:40125
I20260810 18:11:42.601992 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:42.630426 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:42.621703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.23.65.65:37357)
0810 18:11:42.622058 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:42.622067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:42.622090 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:11:42.622138 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:42.622143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:42.622335 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:42.622460 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:42.623112 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.623966 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.624768 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.625173 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.625256 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:42.627454 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:42.627464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:42.627477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:42.627501 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:42.628990 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.629369 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.629374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.629375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.629449 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.629646 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.629647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.629649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.629866 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:42.630015 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:42.630147 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 18:11:42.630266 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":120,"threads_started":1}
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:42.640153 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:42.633238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.23.65.126:46581)
0810 18:11:42.633387 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:42.633390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:42.633405 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:42.633486 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:42.633489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:42.633527 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:42.633590 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:42.634086 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.634579 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.635237 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:42.635454 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:42.635578 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:42.637624 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:42.637627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:42.637630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:42.637657 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:42.638750 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.639117 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.639119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.639120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.639193 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:42.639388 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:42.639390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:42.639391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:42.639635 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:42.639737 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:42.639899 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 18:11:42.639992 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 18:11:42.640542 29901 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098
I20260810 18:11:42.641141 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 30008
I20260810 18:11:42.660538 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 30223
I20260810 18:11:42.676642 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 30455
I20260810 18:11:42.689486 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 29842
2026-08-10T18:11:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2266 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[30700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[30700](info): set up 2 sockets
Aug 10 18:11:42 dist-test-slave-jpj1 krb5kdc[30700](info): commencing operation
krb5kdc: starting...
W20260810 18:11:43.850026 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:43 dist-test-slave-jpj1 krb5kdc[30700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:43.971939 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46049
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:44.082733 30725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:44.082988 30725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:44.083029 30725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:44.086972 30725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:44.087028 30725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:44.087044 30725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:44.087059 30725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:44.087081 30725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:44.092243 30725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:44.093202 30725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:44.094167 30725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:44.102097 30738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:44.102113 30734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:44.102113 30733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:44.102411 30725 server_base.cc:1034] running on GCE node
I20260810 18:11:44.102985 30725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:44.104151 30725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:44.105320 30725 hybrid_clock.cc:648] HybridClock initialized: now 1786385504105302 us; error 26 us; skew 500 ppm
Aug 10 18:11:44 dist-test-slave-jpj1 krb5kdc[30700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:44.108045 30725 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:44.108806 30725 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:44.108872 30725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:44.109035 30725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:44.111009 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "134ed009883c4163b2b1c86aac8ec0c1"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "bb9bc4eadb1d3bb6a4e8254f18e85e05"
server_key_iv: "65d4a1c7f723c9dea50bde9b41548ec7"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.111522 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "134ed009883c4163b2b1c86aac8ec0c1"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "bb9bc4eadb1d3bb6a4e8254f18e85e05"
server_key_iv: "65d4a1c7f723c9dea50bde9b41548ec7"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.115053 30725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 18:11:44.117429 30749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.118599 30725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:11:44.118708 30725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "134ed009883c4163b2b1c86aac8ec0c1"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "bb9bc4eadb1d3bb6a4e8254f18e85e05"
server_key_iv: "65d4a1c7f723c9dea50bde9b41548ec7"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.118809 30725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:44.136997 30725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:44.139765 30725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:44.139951 30725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:44.146692 30725 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:46049
I20260810 18:11:44.146942 30844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:46049 every 8 connection(s)
I20260810 18:11:44.147383 30725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:44.147864 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 30725
I20260810 18:11:44.147991 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:44.148290 23813 external_mini_cluster.cc:1428] Setting key 91b1eec0f137119c8ec20f6532c2742f
I20260810 18:11:44.150533 30847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:44.157502 30847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1: Bootstrap starting.
Aug 10 18:11:44 dist-test-slave-jpj1 krb5kdc[30700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:44.159919 30847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:44.160851 30847 log.cc:824] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:44.163131 30847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1: No bootstrap required, opened a new log
I20260810 18:11:44.164499 30854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.149626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.23.65.126:46049)
0810 18:11:44.150057 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.150077 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.151864 (+ 1787us) server_negotiation.cc:407] Connection header received
0810 18:11:44.152650 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.152674 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.153092 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.153479 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:44.154124 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.155608 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.156447 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.157119 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.157284 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.159663 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:44.159669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:44.159685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:44.159717 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:44.161506 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.161944 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.161949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.161955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.162095 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.162321 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.162324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.162325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.162821 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:44.163008 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.163190 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.163513 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":136,"threads_started":1}
I20260810 18:11:44.166947 30847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134ed009883c4163b2b1c86aac8ec0c1" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46049 } }
I20260810 18:11:44.167240 30847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:44.167304 30847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134ed009883c4163b2b1c86aac8ec0c1, State: Initialized, Role: FOLLOWER
I20260810 18:11:44.167757 30847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134ed009883c4163b2b1c86aac8ec0c1" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46049 } }
I20260810 18:11:44.167892 30847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:44.167948 30847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:44.168042 30847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:44.168982 30847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134ed009883c4163b2b1c86aac8ec0c1" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46049 } }
I20260810 18:11:44.169308 30847 leader_election.cc:302] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 134ed009883c4163b2b1c86aac8ec0c1; no voters:
I20260810 18:11:44.169623 30847 leader_election.cc:288] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:44.169749 30856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:44.169984 30856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [term 1 LEADER]: Becoming Leader. State: Replica: 134ed009883c4163b2b1c86aac8ec0c1, State: Running, Role: LEADER
I20260810 18:11:44.170352 30856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134ed009883c4163b2b1c86aac8ec0c1" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46049 } }
I20260810 18:11:44.170799 30847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:44.171957 30859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 134ed009883c4163b2b1c86aac8ec0c1. Latest consensus state: current_term: 1 leader_uuid: "134ed009883c4163b2b1c86aac8ec0c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134ed009883c4163b2b1c86aac8ec0c1" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46049 } } }
I20260810 18:11:44.172083 30859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:44.172402 30858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "134ed009883c4163b2b1c86aac8ec0c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134ed009883c4163b2b1c86aac8ec0c1" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46049 } } }
I20260810 18:11:44.172541 30858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:44.172871 30869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:44.175781 30869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:44.180997 30869 catalog_manager.cc:1347] Generated new cluster ID: f9bd5db13adb4539b5b2de628ad1bb8e
I20260810 18:11:44.181069 30869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:44.205121 30869 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:44.206105 30869 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:44.213316 30869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1: Generated new TSK 0
I20260810 18:11:44.214196 30869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:11:44.253878 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46049
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:44.388767 30888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:44.389011 30888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:44.389052 30888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:44.393224 30888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:44.393290 30888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:44.393348 30888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:44.398443 30888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:44.399385 30888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:44.400388 30888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:44.409449 30894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:44.409459 30893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:44.409461 30898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:44.409691 30888 server_base.cc:1034] running on GCE node
I20260810 18:11:44.410167 30888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:44.410754 30888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:44.411917 30888 hybrid_clock.cc:648] HybridClock initialized: now 1786385504411899 us; error 28 us; skew 500 ppm
Aug 10 18:11:44 dist-test-slave-jpj1 krb5kdc[30700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:44.414546 30888 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:11:44.415256 30888 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:44.415318 30888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:44.415500 30888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:44.417483 30888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "6397476360c84f45aed3ec690569110d"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "7b28661e1fd6872d4884b845393f4ce5"
server_key_iv: "92ef6868daeaa77cc394d028f4966fdf"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.418011 30888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "6397476360c84f45aed3ec690569110d"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "7b28661e1fd6872d4884b845393f4ce5"
server_key_iv: "92ef6868daeaa77cc394d028f4966fdf"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.421666 30888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 18:11:44.424124 30909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.425347 30888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:44.425444 30888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "6397476360c84f45aed3ec690569110d"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "7b28661e1fd6872d4884b845393f4ce5"
server_key_iv: "92ef6868daeaa77cc394d028f4966fdf"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.425545 30888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:44.447335 30888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:44.450089 30888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:44.450273 30888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:44.450834 30888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:44.451673 30888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:44.451723 30888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.451767 30888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:44.451797 30888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.460162 30888 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:37267
I20260810 18:11:44.460173 31088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:37267 every 8 connection(s)
I20260810 18:11:44.460721 30888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:44.461531 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 30888
I20260810 18:11:44.461658 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:44.461992 23813 external_mini_cluster.cc:1428] Setting key 51024c3435fcad0762ae926f131566cf
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Aug 10 18:11:44 dist-test-slave-jpj1 krb5kdc[30700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:44.479321 30854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.462598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:36111 (local address 127.23.65.126:46049)
0810 18:11:44.462745 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.462750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.463775 (+ 1025us) server_negotiation.cc:407] Connection header received
0810 18:11:44.465087 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.465090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.465146 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.465235 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:44.466753 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.467275 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.468434 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.468669 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.468757 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.476075 (+ 7318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:44.476078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:44.476080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:44.476105 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:44.477341 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.477823 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.477826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.477828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.477904 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.478223 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.478227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.478229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.478421 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:44.478505 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.479013 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.479144 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:11:44.480192 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.463050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46049 (local address 127.23.65.65:36111)
0810 18:11:44.463614 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:44.463670 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:44.464878 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:44.465461 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:44.465470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:44.465945 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:44.466605 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.466626 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.467378 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.467381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.468315 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.468325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.468811 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.468815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.468982 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.472080 (+ 3098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:44.472097 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:44.475902 (+ 3805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:44.477450 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:44.477456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:44.477471 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:44.477716 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:44.477985 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:44.477988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:44.477990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:44.478128 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:44.478521 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:44.478528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:44.478780 (+ 252us) client_negotiation.cc:769] Sending connection context
0810 18:11:44.478938 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 18:11:44.479259 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":158,"threads_started":1}
I20260810 18:11:44.481398 31089 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46049
I20260810 18:11:44.481689 31089 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:44.482256 31089 heartbeater.cc:511] Master 127.23.65.126:46049 requested a full tablet report, sending...
I20260810 18:11:44.483851 30780 ts_manager.cc:194] Registered new tserver with Master: 6397476360c84f45aed3ec690569110d (127.23.65.65:37267)
I20260810 18:11:44.485060 30780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:36111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:11:44.490240 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46049
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:44.600765 31096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:44.601008 31096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:44.601050 31096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:44.605221 31096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:44.605273 31096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:44.605335 31096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:44.610664 31096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:44.611621 31096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:44.612571 31096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:44.621539 31102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:44.621551 31109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:44.621531 31106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:44.621881 31096 server_base.cc:1034] running on GCE node
I20260810 18:11:44.622283 31096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:44.622917 31096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:44.624074 31096 hybrid_clock.cc:648] HybridClock initialized: now 1786385504624047 us; error 36 us; skew 500 ppm
Aug 10 18:11:44 dist-test-slave-jpj1 krb5kdc[30700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:44.626775 31096 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:11:44.627445 31096 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:44.627508 31096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:44.627713 31096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:44.629719 31096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "5d2aba020044430fa767b384d35cc84b"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "57fcb88d38159a7141c5750289c05bc9"
server_key_iv: "ade487ea2523fbc13e7ffeaab35d5254"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.630218 31096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "5d2aba020044430fa767b384d35cc84b"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "57fcb88d38159a7141c5750289c05bc9"
server_key_iv: "ade487ea2523fbc13e7ffeaab35d5254"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.633669 31096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:11:44.635869 31118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.637017 31096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:44.637127 31096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "5d2aba020044430fa767b384d35cc84b"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "57fcb88d38159a7141c5750289c05bc9"
server_key_iv: "ade487ea2523fbc13e7ffeaab35d5254"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.637223 31096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:44.654543 31096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:44.657680 31096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:44.657850 31096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:44.658403 31096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:44.659207 31096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:44.659255 31096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.659296 31096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:44.659339 31096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.667865 31096 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:42703
I20260810 18:11:44.667889 31300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:42703 every 8 connection(s)
I20260810 18:11:44.668416 31096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:44.676533 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 31096
I20260810 18:11:44.676638 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:44.676890 23813 external_mini_cluster.cc:1428] Setting key 7dd692a7123fb05b6bef5f28a3ea71e3
Aug 10 18:11:44 dist-test-slave-jpj1 krb5kdc[30700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:44.681360 30854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.670001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:39197 (local address 127.23.65.126:46049)
0810 18:11:44.670115 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.670118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.670964 (+ 846us) server_negotiation.cc:407] Connection header received
0810 18:11:44.672344 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.672348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.672413 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.672487 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:44.673660 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.674179 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.675288 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.675510 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.675648 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.678012 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:44.678016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:44.678019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:44.678043 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:44.679264 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.679862 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.679865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.679866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.679937 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.680317 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.680320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.680322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.680540 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:44.680630 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.681084 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.681203 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:11:44.682070 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.670330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46049 (local address 127.23.65.66:39197)
0810 18:11:44.670801 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:44.670870 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:44.672130 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:44.672660 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:44.672667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:44.673057 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:44.673513 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.673527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.674299 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.674302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.675160 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.675170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.675660 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.675664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.675841 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.676308 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:44.676324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:44.677858 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:44.679391 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:44.679397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:44.679411 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:44.679761 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:44.680012 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:44.680015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:44.680017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:44.680203 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:44.680640 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:44.680645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:44.680869 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 18:11:44.681021 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 18:11:44.681251 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
I20260810 18:11:44.683102 31302 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46049
I20260810 18:11:44.683331 31302 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:44.683794 31302 heartbeater.cc:511] Master 127.23.65.126:46049 requested a full tablet report, sending...
I20260810 18:11:44.684795 30780 ts_manager.cc:194] Registered new tserver with Master: 5d2aba020044430fa767b384d35cc84b (127.23.65.66:42703)
I20260810 18:11:44.685354 30780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:39197
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:11:44.703841 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46049
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:44.815177 31309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:44.815421 31309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:44.815464 31309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:44.819355 31309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:44.819401 31309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:44.819456 31309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:44.824589 31309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:44.825510 31309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:44.826422 31309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:44.835593 31317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:44.835650 31318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:44.835645 31320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:44.835697 31309 server_base.cc:1034] running on GCE node
I20260810 18:11:44.836400 31309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:44.837006 31309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:44.838166 31309 hybrid_clock.cc:648] HybridClock initialized: now 1786385504838150 us; error 26 us; skew 500 ppm
Aug 10 18:11:44 dist-test-slave-jpj1 krb5kdc[30700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:44.841202 31309 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:11:44.841952 31309 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:44.842017 31309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:44.842190 31309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:44.844193 31309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "1083c56206fe4452b561632e60be4713"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "7504d7a0d017211e6a7dcc361e51c9aa"
server_key_iv: "e1429be491080c8b854701eedc0806f4"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.844708 31309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "1083c56206fe4452b561632e60be4713"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "7504d7a0d017211e6a7dcc361e51c9aa"
server_key_iv: "e1429be491080c8b854701eedc0806f4"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.848178 31309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:44.850615 31327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.851812 31309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:44.851912 31309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "1083c56206fe4452b561632e60be4713"
format_stamp: "Formatted at 2026-08-10 18:11:44 on dist-test-slave-jpj1"
server_key: "7504d7a0d017211e6a7dcc361e51c9aa"
server_key_iv: "e1429be491080c8b854701eedc0806f4"
server_key_version: "encryptionkey@0"
I20260810 18:11:44.851994 31309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:44.869207 31309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:44.872408 31309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:44.872573 31309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:44.873116 31309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:44.873910 31309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:44.873960 31309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.874011 31309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:44.874040 31309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:44.882436 31309 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:41715
I20260810 18:11:44.882478 31512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:41715 every 8 connection(s)
I20260810 18:11:44.883001 31309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:44.889751 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 31309
I20260810 18:11:44.889884 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:44.890195 23813 external_mini_cluster.cc:1428] Setting key 5f2efd8afa3d0b344057e61c347be380
Aug 10 18:11:44 dist-test-slave-jpj1 krb5kdc[30700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:44.896687 30854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.884759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:47567 (local address 127.23.65.126:46049)
0810 18:11:44.884890 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.884893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.885660 (+ 767us) server_negotiation.cc:407] Connection header received
0810 18:11:44.887044 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.887048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.887116 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.887193 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:44.888522 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.889247 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.890761 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.891050 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.891157 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.893670 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:44.893674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:44.893676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:44.893701 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:44.895015 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.895431 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.895433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.895434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.895504 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.895822 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.895824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.895825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.896000 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:44.896076 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.896470 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.896568 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:11:44.897467 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.885057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46049 (local address 127.23.65.67:47567)
0810 18:11:44.885498 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:44.885541 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:44.886841 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:44.887364 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:44.887371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:44.887774 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:44.888364 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.888384 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.889386 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.889388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.890636 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.890648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.891204 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.891209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.891407 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.892038 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:44.892055 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:44.893520 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:44.895105 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:44.895109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:44.895120 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:44.895359 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:44.895614 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:44.895616 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:44.895618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:44.895743 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:44.896074 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:44.896078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:44.896294 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 18:11:44.896439 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 18:11:44.896680 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":124,"threads_started":1}
I20260810 18:11:44.898507 31515 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46049
I20260810 18:11:44.898712 31515 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:44.899122 31515 heartbeater.cc:511] Master 127.23.65.126:46049 requested a full tablet report, sending...
I20260810 18:11:44.900115 30780 ts_manager.cc:194] Registered new tserver with Master: 1083c56206fe4452b561632e60be4713 (127.23.65.67:41715)
I20260810 18:11:44.900645 30780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:47567
I20260810 18:11:44.903797 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:11:44.911986 30854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.905590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.23.65.126:46049)
0810 18:11:44.905729 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.905734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.905856 (+ 122us) server_negotiation.cc:407] Connection header received
0810 18:11:44.905960 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.905962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.906000 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.906054 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:44.906731 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.907216 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.907988 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.908190 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.908261 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.909400 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:44.909403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:44.909405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:44.909427 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:44.910691 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.911000 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.911002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.911003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.911072 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:44.911250 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:44.911252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:44.911253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:44.911454 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:44.911589 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.911737 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.911828 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:11:44.914412 30780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:44.917342 30780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:44.931491 31532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.926306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.23.65.65:37267)
0810 18:11:44.927033 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.927040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.927063 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:11:44.927136 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.927141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.927328 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.927443 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:44.927727 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.928764 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.930309 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.930858 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.930954 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.931103 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.931259 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.931354 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":85,"threads_started":1}
I20260810 18:11:44.931486 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.925618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37267 (local address 127.0.0.1:41014)
0810 18:11:44.926550 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:44.926563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:44.926643 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:44.927448 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:44.927451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:44.927471 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:44.927593 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.927600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.928887 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.928891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.930197 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.930207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.930967 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.930970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.931075 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.931111 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 18:11:44.931215 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 18:11:44.931339 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":101,"thread_start_us":110,"threads_started":1}
I20260810 18:11:44.931931 31528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.925498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:42703 (local address 127.0.0.1:35084)
0810 18:11:44.925951 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:44.926003 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:44.926125 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:44.927001 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:44.927007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:44.927028 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:44.927164 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.927171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.928607 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.928611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.930228 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.930241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.931511 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.931515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.931633 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.931656 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:11:44.931723 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 18:11:44.931789 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":43,"thread_start_us":87,"threads_started":1}
I20260810 18:11:44.932360 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.925644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.23.65.66:42703)
0810 18:11:44.926255 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.926267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.926312 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:11:44.926395 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.926400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.926622 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.926794 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:44.927304 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.928476 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.930372 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.931210 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.931311 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.931447 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.932108 (+ 661us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.932206 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":161,"threads_started":1}
I20260810 18:11:44.932729 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.926115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:41715 (local address 127.0.0.1:59936)
0810 18:11:44.926827 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:44.926840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:44.926919 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:44.928109 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:44.928112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:44.928128 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:44.928232 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.928238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.929654 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.929657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.930919 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.930930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.932312 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.932316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.932412 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.932437 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:11:44.932537 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 18:11:44.932608 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":18,"thread_start_us":89,"threads_started":1}
I20260810 18:11:44.933182 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.926525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.23.65.67:41715)
0810 18:11:44.927752 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.927763 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.927786 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:11:44.927855 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.927859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.927987 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.928112 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:44.928346 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.929491 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.931060 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.932173 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.932274 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.932413 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.932938 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.933032 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1122,"thread_start_us":90,"threads_started":1}
I20260810 18:11:44.934906 31196 tablet_service.cc:1467] Processing CreateTablet for tablet 42077d721fdb42b194f10f34cb204f78 (DEFAULT_TABLE table=test-table [id=f3effcfd164e4aa587e4471f1e3c6de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:44.934809 30990 tablet_service.cc:1467] Processing CreateTablet for tablet 42077d721fdb42b194f10f34cb204f78 (DEFAULT_TABLE table=test-table [id=f3effcfd164e4aa587e4471f1e3c6de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:44.935882 31196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42077d721fdb42b194f10f34cb204f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:44.935900 30990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42077d721fdb42b194f10f34cb204f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:44.935714 31411 tablet_service.cc:1467] Processing CreateTablet for tablet 42077d721fdb42b194f10f34cb204f78 (DEFAULT_TABLE table=test-table [id=f3effcfd164e4aa587e4471f1e3c6de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:44.936606 31411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42077d721fdb42b194f10f34cb204f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:44.942178 31534 tablet_bootstrap.cc:492] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: Bootstrap starting.
I20260810 18:11:44.943805 31537 tablet_bootstrap.cc:492] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d: Bootstrap starting.
I20260810 18:11:44.943806 31538 tablet_bootstrap.cc:492] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: Bootstrap starting.
I20260810 18:11:44.944375 31534 tablet_bootstrap.cc:654] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:44.945380 31534 log.cc:824] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:44.945916 31537 tablet_bootstrap.cc:654] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:44.945915 31538 tablet_bootstrap.cc:654] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:44.947104 31538 log.cc:824] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:44.947104 31537 log.cc:824] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:44.947261 31534 tablet_bootstrap.cc:492] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: No bootstrap required, opened a new log
I20260810 18:11:44.947501 31534 ts_tablet_manager.cc:1397] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:11:44.948856 31538 tablet_bootstrap.cc:492] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: No bootstrap required, opened a new log
I20260810 18:11:44.949038 31538 ts_tablet_manager.cc:1397] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:11:44.949090 31537 tablet_bootstrap.cc:492] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d: No bootstrap required, opened a new log
I20260810 18:11:44.949271 31537 ts_tablet_manager.cc:1397] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:11:44.951356 31534 raft_consensus.cc:348] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } }
I20260810 18:11:44.951702 31534 raft_consensus.cc:374] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:44.951776 31534 raft_consensus.cc:729] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2aba020044430fa767b384d35cc84b, State: Initialized, Role: FOLLOWER
I20260810 18:11:44.951705 31538 raft_consensus.cc:348] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } }
I20260810 18:11:44.951974 31538 raft_consensus.cc:374] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:44.952033 31538 raft_consensus.cc:729] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1083c56206fe4452b561632e60be4713, State: Initialized, Role: FOLLOWER
I20260810 18:11:44.952275 31534 consensus_queue.cc:260] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } }
I20260810 18:11:44.952421 31538 consensus_queue.cc:260] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } }
I20260810 18:11:44.952951 31302 heartbeater.cc:503] Master 127.23.65.126:46049 was elected leader, sending a full tablet report...
I20260810 18:11:44.952967 31537 raft_consensus.cc:348] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } }
I20260810 18:11:44.953197 31537 raft_consensus.cc:374] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:44.953244 31537 raft_consensus.cc:729] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6397476360c84f45aed3ec690569110d, State: Initialized, Role: FOLLOWER
I20260810 18:11:44.953318 31534 ts_tablet_manager.cc:1428] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260810 18:11:44.953332 31538 ts_tablet_manager.cc:1428] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 18:11:44.953631 31515 heartbeater.cc:503] Master 127.23.65.126:46049 was elected leader, sending a full tablet report...
I20260810 18:11:44.953816 31537 consensus_queue.cc:260] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } }
I20260810 18:11:44.954717 31537 ts_tablet_manager.cc:1428] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260810 18:11:44.955086 31089 heartbeater.cc:503] Master 127.23.65.126:46049 was elected leader, sending a full tablet report...
W20260810 18:11:44.961759 31090 tablet.cc:2369] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:44.961750 31545 raft_consensus.cc:482] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:44.961894 31545 raft_consensus.cc:504] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } }
I20260810 18:11:44.962970 31545 leader_election.cc:288] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d2aba020044430fa767b384d35cc84b (127.23.65.66:42703), 6397476360c84f45aed3ec690569110d (127.23.65.65:37267)
I20260810 18:11:44.966337 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.963325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:36565 (local address 127.23.65.66:42703)
0810 18:11:44.963442 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.963447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.963480 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:11:44.963526 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.963545 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.963592 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.963662 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:44.963932 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.964444 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.965444 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.965838 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.965943 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.965996 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.966161 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.966226 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:11:44.966336 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.963169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:42703 (local address 127.23.65.67:36565)
0810 18:11:44.963326 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:44.963341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:44.963419 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:44.963681 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:44.963684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:44.963702 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:44.963837 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.963844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.964552 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.964555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.965323 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.965332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.965947 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.965950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.966054 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.966073 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:11:44.966137 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 18:11:44.966207 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 18:11:44.966527 31532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.963224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:53513 (local address 127.23.65.65:37267)
0810 18:11:44.963334 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.963339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.963787 (+ 448us) server_negotiation.cc:407] Connection header received
0810 18:11:44.963872 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.963874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.963907 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.963966 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:44.964220 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.964684 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.965662 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.966045 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.966147 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.966194 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.966363 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.966429 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:11:44.966547 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.963169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37267 (local address 127.23.65.67:53513)
0810 18:11:44.963702 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:44.963716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:44.963803 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:44.963995 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:44.963998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:44.964017 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:44.964125 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.964130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.964778 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.964781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.965551 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:44.965559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.966147 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.966150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:44.966250 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.966280 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:11:44.966341 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 18:11:44.966407 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":203,"thread_start_us":134,"threads_started":1}
I20260810 18:11:44.966948 31228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42077d721fdb42b194f10f34cb204f78" candidate_uuid: "1083c56206fe4452b561632e60be4713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2aba020044430fa767b384d35cc84b" is_pre_election: true
I20260810 18:11:44.967119 31020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42077d721fdb42b194f10f34cb204f78" candidate_uuid: "1083c56206fe4452b561632e60be4713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6397476360c84f45aed3ec690569110d" is_pre_election: true
I20260810 18:11:44.967243 31228 raft_consensus.cc:2454] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1083c56206fe4452b561632e60be4713 in term 0.
I20260810 18:11:44.967396 31020 raft_consensus.cc:2454] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1083c56206fe4452b561632e60be4713 in term 0.
I20260810 18:11:44.967728 31331 leader_election.cc:302] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1083c56206fe4452b561632e60be4713, 5d2aba020044430fa767b384d35cc84b; no voters:
I20260810 18:11:44.967957 31545 raft_consensus.cc:2781] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:44.968045 31545 raft_consensus.cc:482] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:44.968096 31545 raft_consensus.cc:3037] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:44.969063 31545 raft_consensus.cc:504] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } }
I20260810 18:11:44.969354 31545 leader_election.cc:288] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [CANDIDATE]: Term 1 election: Requested vote from peers 5d2aba020044430fa767b384d35cc84b (127.23.65.66:42703), 6397476360c84f45aed3ec690569110d (127.23.65.65:37267)
I20260810 18:11:44.969677 31228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42077d721fdb42b194f10f34cb204f78" candidate_uuid: "1083c56206fe4452b561632e60be4713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2aba020044430fa767b384d35cc84b"
I20260810 18:11:44.969699 31020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42077d721fdb42b194f10f34cb204f78" candidate_uuid: "1083c56206fe4452b561632e60be4713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6397476360c84f45aed3ec690569110d"
I20260810 18:11:44.969786 31228 raft_consensus.cc:3037] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:44.969810 31020 raft_consensus.cc:3037] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:44.970868 31020 raft_consensus.cc:2454] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1083c56206fe4452b561632e60be4713 in term 1.
I20260810 18:11:44.970877 31228 raft_consensus.cc:2454] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1083c56206fe4452b561632e60be4713 in term 1.
I20260810 18:11:44.971159 31332 leader_election.cc:302] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1083c56206fe4452b561632e60be4713, 6397476360c84f45aed3ec690569110d; no voters:
I20260810 18:11:44.971351 31545 raft_consensus.cc:2781] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:44.971596 31545 raft_consensus.cc:686] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 1 LEADER]: Becoming Leader. State: Replica: 1083c56206fe4452b561632e60be4713, State: Running, Role: LEADER
I20260810 18:11:44.971889 31545 consensus_queue.cc:237] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } }
I20260810 18:11:44.974884 30773 catalog_manager.cc:5495] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1083c56206fe4452b561632e60be4713 (127.23.65.67). New cstate: current_term: 1 leader_uuid: "1083c56206fe4452b561632e60be4713" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1083c56206fe4452b561632e60be4713" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 41715 } health_report { overall_health: HEALTHY } } }
I20260810 18:11:44.992578 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:44.989394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.23.65.67:41715)
0810 18:11:44.989517 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:44.989520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:44.989532 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:11:44.989673 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:44.989676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:44.989725 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:44.989813 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:44.990018 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.990516 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.991316 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:44.991516 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:44.991605 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:44.991869 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:44.992247 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:44.992333 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:44.992381 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 18:11:44.992426 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:11:44.998116 31020 raft_consensus.cc:1261] T 42077d721fdb42b194f10f34cb204f78 P 6397476360c84f45aed3ec690569110d [term 1 FOLLOWER]: Refusing update from remote peer 1083c56206fe4452b561632e60be4713: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:44.998116 31228 raft_consensus.cc:1261] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [term 1 FOLLOWER]: Refusing update from remote peer 1083c56206fe4452b561632e60be4713: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:44.998826 31548 consensus_queue.cc:1035] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2aba020044430fa767b384d35cc84b" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 42703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:44.999011 31545 consensus_queue.cc:1035] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6397476360c84f45aed3ec690569110d" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:45.005703 31552 mvcc.cc:205] Tried to move back new op lower bound from 7317035028465598464 to 7317035028367282176. Current Snapshot: MvccSnapshot[applied={T|T < 7317035028465598464}]
I20260810 18:11:45.008718 31554 mvcc.cc:205] Tried to move back new op lower bound from 7317035028465598464 to 7317035028367282176. Current Snapshot: MvccSnapshot[applied={T|T < 7317035028465598464}]
I20260810 18:11:45.015075 30773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:45.015226 30773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:45.017488 30773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 134ed009883c4163b2b1c86aac8ec0c1: Sending DeleteTablet for 3 replicas of tablet 42077d721fdb42b194f10f34cb204f78
I20260810 18:11:45.018267 31411 tablet_service.cc:1514] Processing DeleteTablet for tablet 42077d721fdb42b194f10f34cb204f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:45 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:59936
I20260810 18:11:45.018234 30990 tablet_service.cc:1514] Processing DeleteTablet for tablet 42077d721fdb42b194f10f34cb204f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:45 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:41014
I20260810 18:11:45.018290 31196 tablet_service.cc:1514] Processing DeleteTablet for tablet 42077d721fdb42b194f10f34cb204f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:45 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:35084
I20260810 18:11:45.019052 31569 tablet_replica.cc:330] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: stopping tablet replica
I20260810 18:11:45.019393 31569 raft_consensus.cc:2229] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:45.019752 31569 raft_consensus.cc:2258] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:45.019763 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 30888
I20260810 18:11:45.020112 31568 tablet_replica.cc:330] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: stopping tablet replica
I20260810 18:11:45.020447 31568 raft_consensus.cc:2229] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:45.020889 31568 raft_consensus.cc:2258] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:45.028801 31568 ts_tablet_manager.cc:1905] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:45.032433 31568 ts_tablet_manager.cc:1918] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:45.032564 31568 log.cc:1196] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/42077d721fdb42b194f10f34cb204f78
I20260810 18:11:45.032992 31568 ts_tablet_manager.cc:1939] T 42077d721fdb42b194f10f34cb204f78 P 1083c56206fe4452b561632e60be4713: Deleting consensus metadata
I20260810 18:11:45.045954 30754 catalog_manager.cc:4841] TS 1083c56206fe4452b561632e60be4713 (127.23.65.67:41715): tablet 42077d721fdb42b194f10f34cb204f78 (table test-table [id=f3effcfd164e4aa587e4471f1e3c6de0]) successfully deleted
I20260810 18:11:45.051666 31569 ts_tablet_manager.cc:1905] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:45.054518 31569 ts_tablet_manager.cc:1918] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:45.054633 31569 log.cc:1196] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/42077d721fdb42b194f10f34cb204f78
I20260810 18:11:45.055059 31569 ts_tablet_manager.cc:1939] T 42077d721fdb42b194f10f34cb204f78 P 5d2aba020044430fa767b384d35cc84b: Deleting consensus metadata
W20260810 18:11:45.056041 30753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:37267 (error 108)
I20260810 18:11:45.056507 30752 catalog_manager.cc:4841] TS 5d2aba020044430fa767b384d35cc84b (127.23.65.66:42703): tablet 42077d721fdb42b194f10f34cb204f78 (table test-table [id=f3effcfd164e4aa587e4471f1e3c6de0]) successfully deleted
I20260810 18:11:45.056891 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 31096
I20260810 18:11:45.057160 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.056853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37267 (local address 127.0.0.1:41028)
0810 18:11:45.056999 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:45.057075 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:37267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 18:11:45.064078 30753 catalog_manager.cc:4584] TS 6397476360c84f45aed3ec690569110d (127.23.65.65:37267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42077d721fdb42b194f10f34cb204f78: Network error: Client connection negotiation failed: client connection to 127.23.65.65:37267: connect: Connection refused (error 111)
I20260810 18:11:45.070932 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 31309
I20260810 18:11:45.088209 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 30725
2026-08-10T18:11:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2397 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T18:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:45Z Disabled control of system clock
I20260810 18:11:45.121348 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43617
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:43617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:11:45.232508 31577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:45.232836 31577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:45.232875 31577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:45.236860 31577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:45.236912 31577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:45.236925 31577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:45.236956 31577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:45.236984 31577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:45.241611 31577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:43617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:45.242609 31577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:45.244042 31577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:45.252345 31587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:45.252419 31586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:45.252393 31592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:45.252498 31577 server_base.cc:1034] running on GCE node
I20260810 18:11:45.253085 31577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:45.254094 31577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:45.255250 31577 hybrid_clock.cc:648] HybridClock initialized: now 1786385505255235 us; error 28 us; skew 500 ppm
I20260810 18:11:45.257041 31577 webserver.cc:460] Webserver started at http://127.23.65.126:46695/ using document root <none> and password file <none>
I20260810 18:11:45.257583 31577 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:45.257654 31577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:45.257858 31577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:45.259873 31577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "e6ace2c78d10ba3e1ea93c0b851a30f3"
server_key_iv: "2519cf66d4ebb89e676a8f3def5f4731"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.260397 31577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "e6ace2c78d10ba3e1ea93c0b851a30f3"
server_key_iv: "2519cf66d4ebb89e676a8f3def5f4731"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.263928 31577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:45.266228 31600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.267479 31577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:45.267637 31577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "e6ace2c78d10ba3e1ea93c0b851a30f3"
server_key_iv: "2519cf66d4ebb89e676a8f3def5f4731"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.267747 31577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:45.272277 31577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:45.272882 31577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:45.273047 31577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:45.279731 31577 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:43617
I20260810 18:11:45.279749 31684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:43617 every 8 connection(s)
I20260810 18:11:45.280663 31577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:45.285089 31686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:45.287093 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 31577
I20260810 18:11:45.287212 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:45.287510 23813 external_mini_cluster.cc:1428] Setting key cc86c8eda73a901434831621af301ad9
I20260810 18:11:45.291124 31686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a: Bootstrap starting.
I20260810 18:11:45.293112 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.288985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.23.65.126:43617)
0810 18:11:45.289660 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:45.289676 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:45.289712 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:11:45.290364 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:45.290382 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:45.290726 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:45.291038 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:45.291895 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:45.291899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:45.291911 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:45.291922 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:45.291983 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:45.292000 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:45.292125 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:45.292199 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 18:11:45.292449 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":159,"threads_started":1}
I20260810 18:11:45.293421 31686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:45.294129 31686 log.cc:824] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:45.295997 31686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a: No bootstrap required, opened a new log
I20260810 18:11:45.299476 31686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43617 } }
I20260810 18:11:45.299728 31686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:45.299794 31686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512a5b9c2414b0b87d8e8d8e1cfc43a, State: Initialized, Role: FOLLOWER
I20260810 18:11:45.300180 31686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43617 } }
I20260810 18:11:45.300315 31686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:45.300379 31686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:45.300477 31686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:45.301302 31686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43617 } }
I20260810 18:11:45.301614 31686 leader_election.cc:302] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7512a5b9c2414b0b87d8e8d8e1cfc43a; no voters:
I20260810 18:11:45.301888 31686 leader_election.cc:288] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:45.301993 31693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:45.302184 31693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [term 1 LEADER]: Becoming Leader. State: Replica: 7512a5b9c2414b0b87d8e8d8e1cfc43a, State: Running, Role: LEADER
I20260810 18:11:45.302489 31693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43617 } }
I20260810 18:11:45.303089 31686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:45.304253 31694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7512a5b9c2414b0b87d8e8d8e1cfc43a. Latest consensus state: current_term: 1 leader_uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43617 } } }
I20260810 18:11:45.304279 31695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7512a5b9c2414b0b87d8e8d8e1cfc43a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43617 } } }
I20260810 18:11:45.304396 31694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:45.304406 31695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:45.304755 31705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:45.307753 31705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:45.313304 31705 catalog_manager.cc:1347] Generated new cluster ID: ba0d0d85f3d247f8b865f873e70f2236
I20260810 18:11:45.313369 31705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:45.333297 31705 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:45.333608 31705 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:45.341210 31705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a: Generated new TSK 0
I20260810 18:11:45.341877 31705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:11:45.344738 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43617
--builtin_ntp_servers=127.23.65.84:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:11:45.484301 31720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:45.484551 31720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:45.484591 31720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:45.488545 31720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:45.488595 31720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:45.488663 31720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:45.493283 31720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:45.494261 31720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:45.495615 31720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:45.504880 31728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:45.504892 31733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:45.504946 31736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:45.505380 31720 server_base.cc:1034] running on GCE node
I20260810 18:11:45.505776 31720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:45.506377 31720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:45.507560 31720 hybrid_clock.cc:648] HybridClock initialized: now 1786385505507503 us; error 41 us; skew 500 ppm
I20260810 18:11:45.509336 31720 webserver.cc:460] Webserver started at http://127.23.65.65:44227/ using document root <none> and password file <none>
I20260810 18:11:45.509922 31720 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:45.509989 31720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:45.510190 31720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:45.512264 31720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "d1aa8514064b449084e7b356e953f780"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "96eded7cbbea7ed6269a7f1b849426c8"
server_key_iv: "e1e783b72acbb67ff844a5e492e98cea"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.512794 31720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "d1aa8514064b449084e7b356e953f780"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "96eded7cbbea7ed6269a7f1b849426c8"
server_key_iv: "e1e783b72acbb67ff844a5e492e98cea"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.516306 31720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:45.518697 31743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.519894 31720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:11:45.519994 31720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "d1aa8514064b449084e7b356e953f780"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "96eded7cbbea7ed6269a7f1b849426c8"
server_key_iv: "e1e783b72acbb67ff844a5e492e98cea"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.520093 31720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:45.524626 31720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:45.525221 31720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:45.525377 31720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:45.525899 31720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:45.526769 31720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:45.526826 31720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.526878 31720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:45.526902 31720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.535176 31720 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:40289
I20260810 18:11:45.535214 31923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:40289 every 8 connection(s)
I20260810 18:11:45.536019 31720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:45.541664 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.537738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:33913 (local address 127.23.65.126:43617)
0810 18:11:45.537880 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:45.537883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:45.538882 (+ 999us) server_negotiation.cc:407] Connection header received
0810 18:11:45.540282 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:45.540285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:45.540324 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:45.540385 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:45.541325 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:45.541327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:45.541329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:45.541340 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:45.541359 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:45.541367 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:45.541434 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:45.541524 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 18:11:45.541559 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:11:45.542078 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 31720
I20260810 18:11:45.542184 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:45.542435 23813 external_mini_cluster.cc:1428] Setting key bcc7c75691c054fc0cb05531aebe0ce2
I20260810 18:11:45.542560 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.538115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43617 (local address 127.23.65.65:33913)
0810 18:11:45.538743 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:45.538782 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:45.540084 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:45.540530 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:45.540537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:45.540621 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:45.541089 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:11:45.541091 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:45.541129 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:11:45.541194 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:45.541427 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:45.541432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:45.541436 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:11:45.541554 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 18:11:45.541747 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":121,"threads_started":1}
I20260810 18:11:45.543805 31924 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43617
I20260810 18:11:45.544027 31924 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:45.544407 31924 heartbeater.cc:511] Master 127.23.65.126:43617 requested a full tablet report, sending...
I20260810 18:11:45.544401 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43617
--builtin_ntp_servers=127.23.65.84:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 18:11:45.545832 31626 ts_manager.cc:194] Registered new tserver with Master: d1aa8514064b449084e7b356e953f780 (127.23.65.65:40289)
W20260810 18:11:45.655306 31928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:45.655568 31928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:45.655628 31928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:45.659602 31928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:45.659652 31928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:45.659708 31928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:45.664332 31928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:45.665279 31928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:45.666553 31928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:45.675771 31938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:45.675855 31936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:45.675940 31940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:45.676015 31928 server_base.cc:1034] running on GCE node
I20260810 18:11:45.676399 31928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:45.676960 31928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:45.678107 31928 hybrid_clock.cc:648] HybridClock initialized: now 1786385505678081 us; error 42 us; skew 500 ppm
I20260810 18:11:45.679891 31928 webserver.cc:460] Webserver started at http://127.23.65.66:44999/ using document root <none> and password file <none>
I20260810 18:11:45.680415 31928 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:45.680476 31928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:45.680672 31928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:45.682628 31928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "d6a32a28a93548c28bba0fe2c5a80bf8"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "47193c33a917da22dea3a8c7470c380b"
server_key_iv: "baa3ac6829164b540ca5a8265cef35c7"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.683142 31928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "d6a32a28a93548c28bba0fe2c5a80bf8"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "47193c33a917da22dea3a8c7470c380b"
server_key_iv: "baa3ac6829164b540ca5a8265cef35c7"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.686578 31928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:11:45.688953 31950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.690089 31928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:45.690193 31928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "d6a32a28a93548c28bba0fe2c5a80bf8"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "47193c33a917da22dea3a8c7470c380b"
server_key_iv: "baa3ac6829164b540ca5a8265cef35c7"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.690287 31928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:45.694669 31928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:45.695228 31928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:45.695379 31928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:45.695950 31928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:45.696813 31928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:45.696861 31928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.696909 31928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:45.696938 31928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.705576 31928 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:39949
I20260810 18:11:45.705602 32118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:39949 every 8 connection(s)
I20260810 18:11:45.706414 31928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:45.711061 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 31928
I20260810 18:11:45.711175 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:45.711423 23813 external_mini_cluster.cc:1428] Setting key 6d331619833df008f48982ed6d261221
I20260810 18:11:45.712582 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.708425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:43375 (local address 127.23.65.126:43617)
0810 18:11:45.708546 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:45.708550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:45.709555 (+ 1005us) server_negotiation.cc:407] Connection header received
0810 18:11:45.710854 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:45.710857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:45.710910 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:45.710988 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:45.712144 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:45.712148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:45.712150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:45.712164 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:45.712188 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:45.712198 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:45.712290 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:45.712414 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 18:11:45.712455 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:11:45.713446 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.708762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43617 (local address 127.23.65.66:43375)
0810 18:11:45.709411 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:45.709451 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:45.710636 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:45.711159 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:45.711168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:45.711278 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:45.711855 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:11:45.711857 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:45.711897 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:11:45.711981 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:45.712290 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:45.712297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:45.712301 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:11:45.712433 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 18:11:45.712625 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260810 18:11:45.713457 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43617
--builtin_ntp_servers=127.23.65.84:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 18:11:45.715941 32119 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43617
I20260810 18:11:45.716216 32119 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:45.716629 32119 heartbeater.cc:511] Master 127.23.65.126:43617 requested a full tablet report, sending...
I20260810 18:11:45.717648 31626 ts_manager.cc:194] Registered new tserver with Master: d6a32a28a93548c28bba0fe2c5a80bf8 (127.23.65.66:39949)
W20260810 18:11:45.824699 32124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:45.824952 32124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:45.824990 32124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:45.828914 32124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:45.828960 32124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:45.829027 32124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:45.833621 32124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:45.834571 32124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:45.835887 32124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:45.845023 32132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:45.845034 32133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:45.845252 32124 server_base.cc:1034] running on GCE node
W20260810 18:11:45.845026 32137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:45.845824 32124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:45.846410 32124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:45.847575 32124 hybrid_clock.cc:648] HybridClock initialized: now 1786385505847556 us; error 34 us; skew 500 ppm
I20260810 18:11:45.849285 32124 webserver.cc:460] Webserver started at http://127.23.65.67:39905/ using document root <none> and password file <none>
I20260810 18:11:45.849872 32124 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:45.849936 32124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:45.850102 32124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:45.852217 32124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "57538a0b392c420e9109a0a4c9dcd77f"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "e09aecce3124ef667a5df6cace3bb284"
server_key_iv: "f20b2819430cc0eea62511c3113b082a"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.852763 32124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "57538a0b392c420e9109a0a4c9dcd77f"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "e09aecce3124ef667a5df6cace3bb284"
server_key_iv: "f20b2819430cc0eea62511c3113b082a"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.856367 32124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:11:45.858780 32147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.859998 32124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:45.860105 32124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "57538a0b392c420e9109a0a4c9dcd77f"
format_stamp: "Formatted at 2026-08-10 18:11:45 on dist-test-slave-jpj1"
server_key: "e09aecce3124ef667a5df6cace3bb284"
server_key_iv: "f20b2819430cc0eea62511c3113b082a"
server_key_version: "encryptionkey@0"
I20260810 18:11:45.860200 32124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:45.864629 32124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:45.865209 32124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:45.865363 32124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:45.865882 32124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:45.866753 32124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:45.866811 32124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.866860 32124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:45.866886 32124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:45.875526 32124 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:34721
I20260810 18:11:45.875584 32330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:34721 every 8 connection(s)
I20260810 18:11:45.876391 32124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:45.879839 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 32124
I20260810 18:11:45.879951 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:45.880203 23813 external_mini_cluster.cc:1428] Setting key cab0c6e41b0ec54c5077dce0e41198ae
I20260810 18:11:45.882221 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.878080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:45647 (local address 127.23.65.126:43617)
0810 18:11:45.878218 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:45.878223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:45.879232 (+ 1009us) server_negotiation.cc:407] Connection header received
0810 18:11:45.880633 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:45.880636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:45.880682 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:45.880756 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:45.881806 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:45.881809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:45.881812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:45.881824 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:45.881845 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:45.881853 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:45.881924 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:45.882048 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 18:11:45.882091 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:11:45.883029 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.878468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43617 (local address 127.23.65.67:45647)
0810 18:11:45.879090 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:45.879129 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:45.880431 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:45.880933 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:45.880940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:45.881028 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:45.881558 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:11:45.881559 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:45.881599 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:11:45.881666 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:45.881920 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:45.881929 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:45.881936 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:11:45.882085 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 18:11:45.882330 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260810 18:11:45.884006 32331 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43617
I20260810 18:11:45.884200 32331 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:45.884555 32331 heartbeater.cc:511] Master 127.23.65.126:43617 requested a full tablet report, sending...
I20260810 18:11:45.885470 31626 ts_manager.cc:194] Registered new tserver with Master: 57538a0b392c420e9109a0a4c9dcd77f (127.23.65.67:34721)
I20260810 18:11:45.893641 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:11:45.896809 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.895489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.23.65.126:43617)
0810 18:11:45.895624 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:45.895628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:45.895787 (+ 159us) server_negotiation.cc:407] Connection header received
0810 18:11:45.895882 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:45.895884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:45.895914 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:45.895977 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:45.896528 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:45.896531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:45.896533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:45.896543 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:45.896562 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:45.896569 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:45.896635 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:45.896680 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 18:11:45.896711 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:11:45.899251 31626 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:45.902194 31626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:45.913528 32351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.910980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:40289 (local address 127.0.0.1:33590)
0810 18:11:45.911517 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:45.911551 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:45.911678 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:45.912096 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:45.912101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:45.912177 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:45.912737 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:11:45.912739 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:45.912760 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:11:45.912825 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:45.913174 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:45.913184 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:45.913189 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:11:45.913311 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 18:11:45.913385 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"spinlock_wait_cycles":85120,"thread_start_us":105,"threads_started":1}
I20260810 18:11:45.913573 32357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.911212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.23.65.65:40289)
0810 18:11:45.911680 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:45.911692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:45.911735 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:11:45.911791 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:45.911796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:45.911948 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:45.912081 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:45.912984 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:45.912990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:45.913004 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:45.913017 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:45.913063 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:45.913098 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:45.913208 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:45.913338 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 18:11:45.913422 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":134,"threads_started":1}
I20260810 18:11:45.913865 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.910664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:34721 (local address 127.0.0.1:41874)
0810 18:11:45.911291 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:45.911342 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:45.911444 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:45.912231 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:45.912234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:45.912250 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:45.913009 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:11:45.913011 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:45.913022 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:11:45.913050 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:45.913643 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:45.913646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:45.913650 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:11:45.913708 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 18:11:45.913743 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":101,"threads_started":1}
I20260810 18:11:45.914276 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.911076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.23.65.67:34721)
0810 18:11:45.911680 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:45.911692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:45.911714 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:11:45.911767 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:45.911773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:45.911948 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:45.912893 (+ 945us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:45.913453 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:45.913459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:45.913470 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:45.913479 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:45.913514 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:45.913537 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:45.914038 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:45.914082 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 18:11:45.914139 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":151,"threads_started":1}
I20260810 18:11:45.914424 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.910417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.23.65.66:39949)
0810 18:11:45.910919 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:45.910933 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:45.911575 (+ 642us) server_negotiation.cc:407] Connection header received
0810 18:11:45.911766 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:45.911772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:45.911935 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:45.912026 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:45.913062 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:45.913067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:45.913076 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:45.913089 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:45.913125 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:45.913155 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:45.913223 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:45.914199 (+ 976us) server_negotiation.cc:299] Negotiation successful
0810 18:11:45.914281 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":152,"threads_started":1}
I20260810 18:11:45.914590 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:45.910306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:39949 (local address 127.0.0.1:52532)
0810 18:11:45.910644 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:45.911455 (+ 811us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:45.911640 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:45.912493 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:45.912496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:45.912509 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:45.912895 (+ 386us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:11:45.912897 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:45.912906 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:11:45.912929 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:45.913295 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:45.913297 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:45.913300 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:11:45.914437 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0810 18:11:45.914472 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"spinlock_wait_cycles":3456,"thread_start_us":80,"threads_started":1}
I20260810 18:11:45.916606 31822 tablet_service.cc:1467] Processing CreateTablet for tablet e05ab2a31ac44aca90615cc40e1de09b (DEFAULT_TABLE table=test-table [id=cf92dc84e74c46da8e201aa9e94b31de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:45.917155 32216 tablet_service.cc:1467] Processing CreateTablet for tablet e05ab2a31ac44aca90615cc40e1de09b (DEFAULT_TABLE table=test-table [id=cf92dc84e74c46da8e201aa9e94b31de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:45.917151 32021 tablet_service.cc:1467] Processing CreateTablet for tablet e05ab2a31ac44aca90615cc40e1de09b (DEFAULT_TABLE table=test-table [id=cf92dc84e74c46da8e201aa9e94b31de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:45.917788 31822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05ab2a31ac44aca90615cc40e1de09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:45.918221 32021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05ab2a31ac44aca90615cc40e1de09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:45.918221 32216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05ab2a31ac44aca90615cc40e1de09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:45.923943 32358 tablet_bootstrap.cc:492] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780: Bootstrap starting.
I20260810 18:11:45.926440 32358 tablet_bootstrap.cc:654] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:45.926627 32360 tablet_bootstrap.cc:492] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: Bootstrap starting.
I20260810 18:11:45.927306 32358 log.cc:824] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:45.927896 32359 tablet_bootstrap.cc:492] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: Bootstrap starting.
I20260810 18:11:45.928771 32360 tablet_bootstrap.cc:654] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:45.929208 32358 tablet_bootstrap.cc:492] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780: No bootstrap required, opened a new log
I20260810 18:11:45.929397 32358 ts_tablet_manager.cc:1397] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:11:45.929680 32360 log.cc:824] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:45.930075 32359 tablet_bootstrap.cc:654] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:45.930770 32359 log.cc:824] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:45.931949 32360 tablet_bootstrap.cc:492] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: No bootstrap required, opened a new log
I20260810 18:11:45.932138 32360 ts_tablet_manager.cc:1397] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:11:45.932961 32359 tablet_bootstrap.cc:492] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: No bootstrap required, opened a new log
I20260810 18:11:45.933143 32359 ts_tablet_manager.cc:1397] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:11:45.933636 32358 raft_consensus.cc:348] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } }
I20260810 18:11:45.933933 32358 raft_consensus.cc:374] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:45.933995 32358 raft_consensus.cc:729] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1aa8514064b449084e7b356e953f780, State: Initialized, Role: FOLLOWER
I20260810 18:11:45.934444 32358 consensus_queue.cc:260] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } }
I20260810 18:11:45.935441 32358 ts_tablet_manager.cc:1428] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:11:45.935575 32360 raft_consensus.cc:348] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } }
I20260810 18:11:45.935809 32360 raft_consensus.cc:374] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:45.935856 31924 heartbeater.cc:503] Master 127.23.65.126:43617 was elected leader, sending a full tablet report...
I20260810 18:11:45.935870 32360 raft_consensus.cc:729] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a32a28a93548c28bba0fe2c5a80bf8, State: Initialized, Role: FOLLOWER
I20260810 18:11:45.936282 32360 consensus_queue.cc:260] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } }
I20260810 18:11:45.937139 32119 heartbeater.cc:503] Master 127.23.65.126:43617 was elected leader, sending a full tablet report...
I20260810 18:11:45.937211 32360 ts_tablet_manager.cc:1428] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260810 18:11:45.937222 32359 raft_consensus.cc:348] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } }
I20260810 18:11:45.937460 32359 raft_consensus.cc:374] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:45.937521 32359 raft_consensus.cc:729] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57538a0b392c420e9109a0a4c9dcd77f, State: Initialized, Role: FOLLOWER
I20260810 18:11:45.937919 32359 consensus_queue.cc:260] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } }
I20260810 18:11:45.938678 32359 ts_tablet_manager.cc:1428] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:11:45.939038 32331 heartbeater.cc:503] Master 127.23.65.126:43617 was elected leader, sending a full tablet report...
W20260810 18:11:45.957316 32120 tablet.cc:2369] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:11:46.037142 31926 tablet.cc:2369] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:46.097985 32366 raft_consensus.cc:482] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:46.098206 32366 raft_consensus.cc:504] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } }
I20260810 18:11:46.099345 32366 leader_election.cc:288] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1aa8514064b449084e7b356e953f780 (127.23.65.65:40289), 57538a0b392c420e9109a0a4c9dcd77f (127.23.65.67:34721)
I20260810 18:11:46.101179 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.099589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:40289 (local address 127.23.65.66:36581)
0810 18:11:46.099716 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:46.099729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:46.099818 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:46.100151 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:46.100154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:46.100168 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:46.100671 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:11:46.100673 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:46.100683 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:11:46.100708 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:46.100964 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:46.100966 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:46.100968 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 18:11:46.101013 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 18:11:46.101047 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 18:11:46.101193 32357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.099753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:36581 (local address 127.23.65.65:40289)
0810 18:11:46.099917 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:46.099921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:46.099939 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:11:46.099986 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:46.099989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:46.100031 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:46.100103 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:46.100841 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:46.100844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:46.100846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:46.100858 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:46.100882 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:46.100892 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:46.100986 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:46.101014 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 18:11:46.101057 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 18:11:46.101311 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.099716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:52767 (local address 127.23.65.67:34721)
0810 18:11:46.099868 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:46.099873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:46.100198 (+ 325us) server_negotiation.cc:407] Connection header received
0810 18:11:46.100276 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:46.100278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:46.100311 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:46.100370 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:46.100939 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:46.100942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:46.100945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:46.100957 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:46.100976 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:46.100986 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:46.101068 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:46.101161 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 18:11:46.101199 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:11:46.101450 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.099622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:34721 (local address 127.23.65.66:52767)
0810 18:11:46.100107 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:46.100120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:46.100202 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:46.100406 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:46.100409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:46.100424 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:46.100792 (+ 368us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:11:46.100794 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:46.100805 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:11:46.100830 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:46.101073 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:46.101076 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:46.101079 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:11:46.101149 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:11:46.101177 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":148,"thread_start_us":119,"threads_started":1}
I20260810 18:11:46.101797 31855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05ab2a31ac44aca90615cc40e1de09b" candidate_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1aa8514064b449084e7b356e953f780" is_pre_election: true
I20260810 18:11:46.102111 31855 raft_consensus.cc:2454] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a32a28a93548c28bba0fe2c5a80bf8 in term 0.
I20260810 18:11:46.102174 32249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05ab2a31ac44aca90615cc40e1de09b" candidate_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57538a0b392c420e9109a0a4c9dcd77f" is_pre_election: true
I20260810 18:11:46.102455 32249 raft_consensus.cc:2454] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a32a28a93548c28bba0fe2c5a80bf8 in term 0.
I20260810 18:11:46.102613 31953 leader_election.cc:302] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1aa8514064b449084e7b356e953f780, d6a32a28a93548c28bba0fe2c5a80bf8; no voters:
I20260810 18:11:46.102888 32366 raft_consensus.cc:2781] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:46.102973 32366 raft_consensus.cc:482] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:46.103024 32366 raft_consensus.cc:3037] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:46.104197 32366 raft_consensus.cc:504] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } }
I20260810 18:11:46.104528 32366 leader_election.cc:288] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [CANDIDATE]: Term 1 election: Requested vote from peers d1aa8514064b449084e7b356e953f780 (127.23.65.65:40289), 57538a0b392c420e9109a0a4c9dcd77f (127.23.65.67:34721)
I20260810 18:11:46.104836 32249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05ab2a31ac44aca90615cc40e1de09b" candidate_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57538a0b392c420e9109a0a4c9dcd77f"
I20260810 18:11:46.104833 31855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05ab2a31ac44aca90615cc40e1de09b" candidate_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1aa8514064b449084e7b356e953f780"
I20260810 18:11:46.104955 31855 raft_consensus.cc:3037] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:46.104955 32249 raft_consensus.cc:3037] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:46.106210 32249 raft_consensus.cc:2454] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a32a28a93548c28bba0fe2c5a80bf8 in term 1.
I20260810 18:11:46.106210 31855 raft_consensus.cc:2454] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a32a28a93548c28bba0fe2c5a80bf8 in term 1.
I20260810 18:11:46.106489 31955 leader_election.cc:302] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57538a0b392c420e9109a0a4c9dcd77f, d6a32a28a93548c28bba0fe2c5a80bf8; no voters:
I20260810 18:11:46.106662 32366 raft_consensus.cc:2781] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:46.106873 32366 raft_consensus.cc:686] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 1 LEADER]: Becoming Leader. State: Replica: d6a32a28a93548c28bba0fe2c5a80bf8, State: Running, Role: LEADER
I20260810 18:11:46.107321 32366 consensus_queue.cc:237] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } }
I20260810 18:11:46.110526 31626 catalog_manager.cc:5495] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6a32a28a93548c28bba0fe2c5a80bf8 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a32a28a93548c28bba0fe2c5a80bf8" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 } health_report { overall_health: UNKNOWN } } }
W20260810 18:11:46.127315 32332 tablet.cc:2369] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:46.148168 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.146856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.23.65.66:39949)
0810 18:11:46.146986 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:46.146989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:46.147005 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:11:46.147086 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:46.147088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:46.147147 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:46.147216 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:46.147871 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:46.147874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:46.147876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:46.147887 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:46.147907 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:46.147915 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:46.147986 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:46.148025 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 18:11:46.148058 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:11:46.154237 31855 raft_consensus.cc:1261] T e05ab2a31ac44aca90615cc40e1de09b P d1aa8514064b449084e7b356e953f780 [term 1 FOLLOWER]: Refusing update from remote peer d6a32a28a93548c28bba0fe2c5a80bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:46.154237 32249 raft_consensus.cc:1261] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [term 1 FOLLOWER]: Refusing update from remote peer d6a32a28a93548c28bba0fe2c5a80bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:46.155054 32373 consensus_queue.cc:1035] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1aa8514064b449084e7b356e953f780" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:46.155308 32366 consensus_queue.cc:1035] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57538a0b392c420e9109a0a4c9dcd77f" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:46.163751 32377 mvcc.cc:205] Tried to move back new op lower bound from 7317035033199538176 to 7317035033018019840. Current Snapshot: MvccSnapshot[applied={T|T < 7317035033199538176}]
I20260810 18:11:46.172067 31625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:46.172276 31625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:46.174563 31625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7512a5b9c2414b0b87d8e8d8e1cfc43a: Sending DeleteTablet for 3 replicas of tablet e05ab2a31ac44aca90615cc40e1de09b
I20260810 18:11:46.175287 32216 tablet_service.cc:1514] Processing DeleteTablet for tablet e05ab2a31ac44aca90615cc40e1de09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:46 UTC) from {username='slave'} at 127.0.0.1:41874
I20260810 18:11:46.175367 31822 tablet_service.cc:1514] Processing DeleteTablet for tablet e05ab2a31ac44aca90615cc40e1de09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:46 UTC) from {username='slave'} at 127.0.0.1:33590
I20260810 18:11:46.175843 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 31720
I20260810 18:11:46.191023 32021 tablet_service.cc:1514] Processing DeleteTablet for tablet e05ab2a31ac44aca90615cc40e1de09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:46 UTC) from {username='slave'} at 127.0.0.1:52532
I20260810 18:11:46.200436 32392 tablet_replica.cc:330] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: stopping tablet replica
I20260810 18:11:46.200855 32392 raft_consensus.cc:2229] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:46.201184 32392 raft_consensus.cc:2258] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:46.203754 32394 tablet_replica.cc:330] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: stopping tablet replica
I20260810 18:11:46.204075 32394 raft_consensus.cc:2229] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:46.204495 32394 raft_consensus.cc:2258] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:46.205675 32392 ts_tablet_manager.cc:1905] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:46.209451 32392 ts_tablet_manager.cc:1918] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:46.209602 32392 log.cc:1196] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/e05ab2a31ac44aca90615cc40e1de09b
I20260810 18:11:46.210016 32392 ts_tablet_manager.cc:1939] T e05ab2a31ac44aca90615cc40e1de09b P 57538a0b392c420e9109a0a4c9dcd77f: Deleting consensus metadata
I20260810 18:11:46.211800 31604 catalog_manager.cc:4841] TS 57538a0b392c420e9109a0a4c9dcd77f (127.23.65.67:34721): tablet e05ab2a31ac44aca90615cc40e1de09b (table test-table [id=cf92dc84e74c46da8e201aa9e94b31de]) successfully deleted
I20260810 18:11:46.216852 32394 ts_tablet_manager.cc:1905] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:46.219753 32394 ts_tablet_manager.cc:1918] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:46.219879 32394 log.cc:1196] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/e05ab2a31ac44aca90615cc40e1de09b
I20260810 18:11:46.220208 32394 ts_tablet_manager.cc:1939] T e05ab2a31ac44aca90615cc40e1de09b P d6a32a28a93548c28bba0fe2c5a80bf8: Deleting consensus metadata
W20260810 18:11:46.221227 31602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:40289 (error 108)
I20260810 18:11:46.221319 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 31928
I20260810 18:11:46.224889 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.224507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:40289 (local address 127.0.0.1:33604)
0810 18:11:46.224658 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:46.224758 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:40289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 18:11:46.226437 31602 catalog_manager.cc:4584] TS d1aa8514064b449084e7b356e953f780 (127.23.65.65:40289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e05ab2a31ac44aca90615cc40e1de09b: Network error: Client connection negotiation failed: client connection to 127.23.65.65:40289: connect: Connection refused (error 111)
I20260810 18:11:46.231902 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 32124
I20260810 18:11:46.232345 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.232041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:39949 (local address 127.0.0.1:52550)
0810 18:11:46.232213 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:46.232261 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.66:39949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 18:11:46.247298 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 31577
2026-08-10T18:11:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1160 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 18:11:46.273733 23813 mini_oidc.cc:152] Starting JWKS server
I20260810 18:11:46.275524 23813 webserver.cc:460] Webserver started at https://127.0.0.1:39139/ using document root <none> and password file <none>
I20260810 18:11:46.275745 23813 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:11:46.276023 23813 webserver.cc:460] Webserver started at http://127.0.0.1:46703/ using document root <none> and password file <none>
2026-08-10T18:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:46Z Disabled control of system clock
I20260810 18:11:46.285580 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:41181
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:41181
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39139/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/testchainca.cert with env {}
W20260810 18:11:46.394793 32405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:46.395045 32405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:46.395082 32405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:46.398934 32405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:46.398983 32405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:46.398996 32405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:46.399010 32405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:46.399029 32405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:11:46.399047 32405 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:11:46.399060 32405 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39139/jwks.json
I20260810 18:11:46.404256 32405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:41181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39139/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:46.405203 32405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:46.406119 32405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:46.414065 32414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:46.414055 32413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:46.414181 32405 server_base.cc:1034] running on GCE node
W20260810 18:11:46.414055 32416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:46.414873 32405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:46.415946 32405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:46.417131 32405 hybrid_clock.cc:648] HybridClock initialized: now 1786385506417100 us; error 31 us; skew 500 ppm
I20260810 18:11:46.418933 32405 webserver.cc:460] Webserver started at http://127.23.65.126:32853/ using document root <none> and password file <none>
I20260810 18:11:46.419485 32405 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:46.419589 32405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:46.419804 32405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:46.421777 32405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "02c0f55b4b754bb19dffea7d60a8e641"
format_stamp: "Formatted at 2026-08-10 18:11:46 on dist-test-slave-jpj1"
server_key: "194844624a42d6fc243848626a5b7437"
server_key_iv: "9d6d955e89f66ecb7d1910b7feebbbaa"
server_key_version: "encryptionkey@0"
I20260810 18:11:46.422322 32405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "02c0f55b4b754bb19dffea7d60a8e641"
format_stamp: "Formatted at 2026-08-10 18:11:46 on dist-test-slave-jpj1"
server_key: "194844624a42d6fc243848626a5b7437"
server_key_iv: "9d6d955e89f66ecb7d1910b7feebbbaa"
server_key_version: "encryptionkey@0"
I20260810 18:11:46.425781 32405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:11:46.428051 32423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:46.429241 32405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:46.429345 32405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "02c0f55b4b754bb19dffea7d60a8e641"
format_stamp: "Formatted at 2026-08-10 18:11:46 on dist-test-slave-jpj1"
server_key: "194844624a42d6fc243848626a5b7437"
server_key_iv: "9d6d955e89f66ecb7d1910b7feebbbaa"
server_key_version: "encryptionkey@0"
I20260810 18:11:46.429447 32405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:46.443253 32405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:46.443987 32405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:46.444176 32405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:46.451797 32405 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:41181
I20260810 18:11:46.451810 32510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:41181 every 8 connection(s)
I20260810 18:11:46.452771 32405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:46.456091 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:46.460589 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 32405
I20260810 18:11:46.460717 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:46.461006 23813 external_mini_cluster.cc:1428] Setting key 33626e486068fcd60e12624840715e1d
I20260810 18:11:46.462036 32511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641: Bootstrap starting.
I20260810 18:11:46.464716 32511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:46.465615 32511 log.cc:824] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:46.467705 32511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641: No bootstrap required, opened a new log
I20260810 18:11:46.471199 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.462429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.23.65.126:41181)
0810 18:11:46.463113 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:46.463134 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:46.463178 (+ 44us) server_negotiation.cc:407] Connection header received
0810 18:11:46.464008 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:46.464032 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:46.464548 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:46.464913 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:46.465373 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.466586 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.467490 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.468185 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.468354 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:46.469117 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:46.469124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:46.469141 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:46.469663 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:46.469734 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:46.469760 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:46.469859 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:46.469955 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 18:11:46.470222 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260810 18:11:46.472100 32511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c0f55b4b754bb19dffea7d60a8e641" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 41181 } }
I20260810 18:11:46.472317 32511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:46.472375 32511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c0f55b4b754bb19dffea7d60a8e641, State: Initialized, Role: FOLLOWER
I20260810 18:11:46.472806 32511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c0f55b4b754bb19dffea7d60a8e641" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 41181 } }
I20260810 18:11:46.472961 32511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:46.473024 32511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:46.473116 32511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:46.474164 32511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c0f55b4b754bb19dffea7d60a8e641" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 41181 } }
I20260810 18:11:46.474499 32511 leader_election.cc:302] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02c0f55b4b754bb19dffea7d60a8e641; no voters:
I20260810 18:11:46.474828 32511 leader_election.cc:288] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:46.474855 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:46.475193 32519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:46.475449 32519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [term 1 LEADER]: Becoming Leader. State: Replica: 02c0f55b4b754bb19dffea7d60a8e641, State: Running, Role: LEADER
I20260810 18:11:46.476137 32511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:46.476645 32519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c0f55b4b754bb19dffea7d60a8e641" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 41181 } }
I20260810 18:11:46.477856 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02c0f55b4b754bb19dffea7d60a8e641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c0f55b4b754bb19dffea7d60a8e641" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 41181 } } }
I20260810 18:11:46.477878 32522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02c0f55b4b754bb19dffea7d60a8e641. Latest consensus state: current_term: 1 leader_uuid: "02c0f55b4b754bb19dffea7d60a8e641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c0f55b4b754bb19dffea7d60a8e641" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 41181 } } }
I20260810 18:11:46.478008 32522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:46.478008 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641 [sys.catalog]: This master's current role is: LEADER
W20260810 18:11:46.481180 32547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:46.481271 32547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:46.481371 32548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:46.482229 32548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:46.487702 32548 catalog_manager.cc:1347] Generated new cluster ID: b0e3e3bcd88749efb2f0b66fca75a33f
I20260810 18:11:46.487779 32548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:46.494830 32548 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:46.495776 32548 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:46.501168 32548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02c0f55b4b754bb19dffea7d60a8e641: Generated new TSK 0
I20260810 18:11:46.501812 32548 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 18:11:46.532070 32559 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 18:11:46.532244 32559 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 18:11:46.532786 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.514722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.23.65.126:41181)
0810 18:11:46.514854 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:46.514859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:46.515078 (+ 219us) server_negotiation.cc:407] Connection header received
0810 18:11:46.515297 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:46.515299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:46.515345 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:46.515425 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:11:46.515710 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.516255 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.517479 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.517697 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.517767 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:46.518200 (+ 433us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:11:46.532426 (+ 14226us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 18:11:46.532669 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36034: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 18:11:46.532869 32516 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36034: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 18:11:46.534662 32558 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:46.514642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:41181 (local address 127.0.0.1:36034)
0810 18:11:46.514993 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:46.515009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:46.515143 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:46.515457 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:46.515460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:46.515468 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:11:46.515613 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:46.515621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.516367 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.516371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:46.517373 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:46.517384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.517875 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.517879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:46.517970 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:46.518026 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:11:46.532668 (+ 14642us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 18:11:46.533054 (+ 386us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.65.126:41181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.65.126:41181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 18:11:46.535763 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 32405
2026-08-10T18:11:46Z chronyd exiting
I20260810 18:11:46.578013 23813 test_util.cc:175] -----------------------------------------------
I20260810 18:11:46.578101 23813 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385478759079-23813-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 18:11:46.579247 23813 mini_oidc.cc:152] Starting JWKS server
I20260810 18:11:46.580138 23813 webserver.cc:460] Webserver started at https://127.0.0.1:38259/ using document root <none> and password file <none>
I20260810 18:11:46.580237 23813 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:11:46.580487 23813 webserver.cc:460] Webserver started at http://127.0.0.1:43943/ using document root <none> and password file <none>
2026-08-10T18:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:46Z Disabled control of system clock
I20260810 18:11:46.590111 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:39111
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:39111
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38259/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/testchainca.cert with env {}
W20260810 18:11:46.698879 32570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:46.699144 32570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:46.699182 32570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:46.703040 32570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:46.703094 32570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:46.703109 32570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:46.703125 32570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:46.703143 32570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:11:46.703157 32570 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:11:46.703176 32570 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38259/jwks.json
I20260810 18:11:46.708468 32570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:39111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38259/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:46.709486 32570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:46.710453 32570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:46.718828 32576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:46.718883 32570 server_base.cc:1034] running on GCE node
W20260810 18:11:46.718845 32580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:46.720877 32578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:46.721468 32570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:46.722584 32570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:46.723752 32570 hybrid_clock.cc:648] HybridClock initialized: now 1786385506723733 us; error 28 us; skew 500 ppm
I20260810 18:11:46.725526 32570 webserver.cc:460] Webserver started at http://127.23.65.126:39695/ using document root <none> and password file <none>
I20260810 18:11:46.726087 32570 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:46.726156 32570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:46.726346 32570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:46.728410 32570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "6478917083994668bca02f145ed7f20b"
format_stamp: "Formatted at 2026-08-10 18:11:46 on dist-test-slave-jpj1"
server_key: "d1db08e1eba89145680e2a1f5d9f4f35"
server_key_iv: "c35e239cc6d83cadfb3154c963b5578c"
server_key_version: "encryptionkey@0"
I20260810 18:11:46.728956 32570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "6478917083994668bca02f145ed7f20b"
format_stamp: "Formatted at 2026-08-10 18:11:46 on dist-test-slave-jpj1"
server_key: "d1db08e1eba89145680e2a1f5d9f4f35"
server_key_iv: "c35e239cc6d83cadfb3154c963b5578c"
server_key_version: "encryptionkey@0"
I20260810 18:11:46.732496 32570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:46.734741 32589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:46.735965 32570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:46.736079 32570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "6478917083994668bca02f145ed7f20b"
format_stamp: "Formatted at 2026-08-10 18:11:46 on dist-test-slave-jpj1"
server_key: "d1db08e1eba89145680e2a1f5d9f4f35"
server_key_iv: "c35e239cc6d83cadfb3154c963b5578c"
server_key_version: "encryptionkey@0"
I20260810 18:11:46.736172 32570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:46.748000 32570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:46.748636 32570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:46.748805 32570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:46.755733 32570 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:39111
I20260810 18:11:46.755754 32678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:39111 every 8 connection(s)
I20260810 18:11:46.756634 32570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:46.759684 32679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:46.765033 32679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b: Bootstrap starting.
I20260810 18:11:46.765705 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 32570
I20260810 18:11:46.765831 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:46.766108 23813 external_mini_cluster.cc:1428] Setting key fbf122cbc182bb6f4224003577b5651f
I20260810 18:11:46.768283 32679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:46.768998 32679 log.cc:824] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:46.770738 32679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b: No bootstrap required, opened a new log
I20260810 18:11:46.774701 32679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6478917083994668bca02f145ed7f20b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39111 } }
I20260810 18:11:46.774914 32679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:46.774972 32679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6478917083994668bca02f145ed7f20b, State: Initialized, Role: FOLLOWER
I20260810 18:11:46.775074 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.767296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.23.65.126:39111)
0810 18:11:46.767911 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:46.767925 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:46.767961 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:11:46.768627 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:46.768647 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:46.769042 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:46.769401 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:46.769834 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.770757 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.771951 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.772423 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.772538 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:46.773363 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:46.773368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:46.773380 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:46.773749 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:46.773804 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:46.773823 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:46.773899 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:46.774053 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 18:11:46.774312 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260810 18:11:46.775489 32679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6478917083994668bca02f145ed7f20b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39111 } }
I20260810 18:11:46.775676 32679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:46.775740 32679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:46.775833 32679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:46.776685 32679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6478917083994668bca02f145ed7f20b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39111 } }
I20260810 18:11:46.777037 32679 leader_election.cc:302] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6478917083994668bca02f145ed7f20b; no voters:
I20260810 18:11:46.777343 32679 leader_election.cc:288] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:46.777444 32684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:46.777653 32684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [term 1 LEADER]: Becoming Leader. State: Replica: 6478917083994668bca02f145ed7f20b, State: Running, Role: LEADER
I20260810 18:11:46.777899 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:46.777980 32684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6478917083994668bca02f145ed7f20b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39111 } }
I20260810 18:11:46.778579 32679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:46.779444 32686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6478917083994668bca02f145ed7f20b. Latest consensus state: current_term: 1 leader_uuid: "6478917083994668bca02f145ed7f20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6478917083994668bca02f145ed7f20b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39111 } } }
I20260810 18:11:46.779592 32686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:46.779932 32685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6478917083994668bca02f145ed7f20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6478917083994668bca02f145ed7f20b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39111 } } }
I20260810 18:11:46.780030 32685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:46.780309 32698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:46.782963 32698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:46.788185 32698 catalog_manager.cc:1347] Generated new cluster ID: efc9cba4eccc4cf89ad4675dd1ceb85d
I20260810 18:11:46.788254 32698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:46.805182 32698 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:46.806077 32698 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:46.810199 32698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6478917083994668bca02f145ed7f20b: Generated new TSK 0
I20260810 18:11:46.810786 32698 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 18:11:46.834074 32719 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 18:11:46.834146 32719 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 18:11:46.834806 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:46.819457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.23.65.126:39111)
0810 18:11:46.819599 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:46.819603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:46.819741 (+ 138us) server_negotiation.cc:407] Connection header received
0810 18:11:46.819873 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:46.819876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:46.819931 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:46.820039 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:11:46.820215 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.821039 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.821859 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.822206 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.822298 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:46.822539 (+ 241us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:11:46.834428 (+ 11889us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 18:11:46.834663 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 18:11:46.834848 32717 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:46.819369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:39111 (local address 127.0.0.1:51408)
0810 18:11:46.819665 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:46.819677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:46.819770 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:46.820034 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:46.820036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:46.820041 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:11:46.820132 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:46.820137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.821134 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.821136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:46.821753 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:46.821760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:46.822299 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:46.822301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:46.822371 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:46.822393 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:11:46.834582 (+ 12189us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 18:11:46.834707 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.65.126:39111: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":96,"threads_started":1}
W20260810 18:11:46.834870 32682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260810 18:11:46.835561 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 32570
2026-08-10T18:11:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 18:11:46.885629 23813 mini_oidc.cc:152] Starting JWKS server
I20260810 18:11:46.886466 23813 webserver.cc:460] Webserver started at https://127.0.0.1:46701/ using document root <none> and password file <none>
I20260810 18:11:46.886569 23813 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:11:46.886821 23813 webserver.cc:460] Webserver started at http://127.0.0.1:38213/ using document root <none> and password file <none>
2026-08-10T18:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:46Z Disabled control of system clock
I20260810 18:11:46.896332 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38821
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:38821
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46701/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 18:11:47.005203 32728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:47.005473 32728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:47.005527 32728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:47.009460 32728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:47.009513 32728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:47.009534 32728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:47.009555 32728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:47.009600 32728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:11:47.009630 32728 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:11:47.009658 32728 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46701/jwks.json
I20260810 18:11:47.014324 32728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:38821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46701/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:47.015360 32728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:47.016744 32728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:47.025197 32735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:47.025293 32728 server_base.cc:1034] running on GCE node
W20260810 18:11:47.025352 32736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:47.025228 32741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:47.026067 32728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:47.027191 32728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:47.028360 32728 hybrid_clock.cc:648] HybridClock initialized: now 1786385507028340 us; error 33 us; skew 500 ppm
I20260810 18:11:47.030267 32728 webserver.cc:460] Webserver started at http://127.23.65.126:44535/ using document root <none> and password file <none>
I20260810 18:11:47.030834 32728 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:47.030900 32728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:47.031124 32728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:47.033259 32728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "6d0ae1e613374285870a39fbb4c33fd7"
format_stamp: "Formatted at 2026-08-10 18:11:47 on dist-test-slave-jpj1"
server_key: "27e5c20a25d971bade924e0feb41041c"
server_key_iv: "762e78dbc3c978ee804ca39f7a21b2b7"
server_key_version: "encryptionkey@0"
I20260810 18:11:47.033843 32728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "6d0ae1e613374285870a39fbb4c33fd7"
format_stamp: "Formatted at 2026-08-10 18:11:47 on dist-test-slave-jpj1"
server_key: "27e5c20a25d971bade924e0feb41041c"
server_key_iv: "762e78dbc3c978ee804ca39f7a21b2b7"
server_key_version: "encryptionkey@0"
I20260810 18:11:47.037655 32728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 18:11:47.040086 32748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:47.041260 32728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:47.041395 32728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "6d0ae1e613374285870a39fbb4c33fd7"
format_stamp: "Formatted at 2026-08-10 18:11:47 on dist-test-slave-jpj1"
server_key: "27e5c20a25d971bade924e0feb41041c"
server_key_iv: "762e78dbc3c978ee804ca39f7a21b2b7"
server_key_version: "encryptionkey@0"
I20260810 18:11:47.041499 32728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:47.081503 32728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:47.082283 32728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:47.082475 32728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:47.089648 32728 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:38821
I20260810 18:11:47.089651 371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:38821 every 8 connection(s)
I20260810 18:11:47.090732 32728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:47.091991 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 32728
I20260810 18:11:47.092118 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:47.092373 23813 external_mini_cluster.cc:1428] Setting key 0dcfe8200ff35b90f4b86425c16b2e36
I20260810 18:11:47.094401 372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:47.101361 372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7: Bootstrap starting.
I20260810 18:11:47.103075 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:47.093772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.23.65.126:38821)
0810 18:11:47.094466 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:47.094487 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:47.094530 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:11:47.095403 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:47.095428 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:47.095973 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:47.096343 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:47.096797 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.098289 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.099240 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.099957 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.100119 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:47.100953 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:47.100959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:47.100976 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:47.101481 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:47.101556 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:47.101583 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:47.101680 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:47.101843 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 18:11:47.102116 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260810 18:11:47.104161 372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:47.104911 372 log.cc:824] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:47.105988 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:47.106813 372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7: No bootstrap required, opened a new log
I20260810 18:11:47.110229 372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0ae1e613374285870a39fbb4c33fd7" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38821 } }
I20260810 18:11:47.110453 372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:47.110502 372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d0ae1e613374285870a39fbb4c33fd7, State: Initialized, Role: FOLLOWER
I20260810 18:11:47.110891 372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0ae1e613374285870a39fbb4c33fd7" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38821 } }
I20260810 18:11:47.111018 372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:47.111085 372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:47.111181 372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:47.112116 372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0ae1e613374285870a39fbb4c33fd7" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38821 } }
I20260810 18:11:47.112447 372 leader_election.cc:302] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d0ae1e613374285870a39fbb4c33fd7; no voters:
I20260810 18:11:47.112730 372 leader_election.cc:288] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:47.112849 379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:47.113106 379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [term 1 LEADER]: Becoming Leader. State: Replica: 6d0ae1e613374285870a39fbb4c33fd7, State: Running, Role: LEADER
I20260810 18:11:47.113400 379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0ae1e613374285870a39fbb4c33fd7" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38821 } }
I20260810 18:11:47.113844 372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:47.114972 380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d0ae1e613374285870a39fbb4c33fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0ae1e613374285870a39fbb4c33fd7" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38821 } } }
I20260810 18:11:47.115113 380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:47.115396 381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d0ae1e613374285870a39fbb4c33fd7. Latest consensus state: current_term: 1 leader_uuid: "6d0ae1e613374285870a39fbb4c33fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0ae1e613374285870a39fbb4c33fd7" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38821 } } }
I20260810 18:11:47.115494 381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:47.115626 388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:47.118666 388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:47.124403 388 catalog_manager.cc:1347] Generated new cluster ID: 49189fd0599a48d39a30141ebf8ba6be
I20260810 18:11:47.124466 388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:47.136338 388 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:47.137291 388 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:47.161407 388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d0ae1e613374285870a39fbb4c33fd7: Generated new TSK 0
I20260810 18:11:47.162101 388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:11:47.173247 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:47.168894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.23.65.126:38821)
0810 18:11:47.169003 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:47.169008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:47.169188 (+ 180us) server_negotiation.cc:407] Connection header received
0810 18:11:47.169347 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:47.169350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:47.169394 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:47.169484 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:11:47.169780 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.170287 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.171122 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.171333 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.171419 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:47.171741 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:11:47.172892 (+ 1151us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 18:11:47.173119 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 18:11:47.173324 376 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 18:11:47.173350 405 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:47.168826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38821 (local address 127.0.0.1:49234)
0810 18:11:47.169107 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:47.169119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:47.169224 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:47.169493 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:47.169497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:47.169505 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:11:47.169667 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:47.169674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.170393 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.170397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:47.171014 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:47.171022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.171441 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.171445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:47.171555 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:47.171585 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:11:47.173049 (+ 1464us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 18:11:47.173184 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.65.126:38821: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":116,"threads_started":1}
I20260810 18:11:47.173885 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 32728
E20260810 18:11:47.190555 406 webserver.cc:533] Webserver: sslize failed:
E20260810 18:11:47.190649 406 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T18:11:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T18:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:47Z Disabled control of system clock
I20260810 18:11:47.231047 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46601
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46601
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/testchainca.cert with env {}
W20260810 18:11:47.342453 411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:47.342723 411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:47.342763 411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:47.346720 411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:47.346772 411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:47.346788 411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:47.346803 411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:47.346817 411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:11:47.346833 411 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:11:47.346850 411 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 18:11:47.352037 411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:47.353053 411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:47.354014 411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:47.362233 419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:47.362246 423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:47.362241 420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:47.362300 411 server_base.cc:1034] running on GCE node
I20260810 18:11:47.362998 411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:47.364108 411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:47.365255 411 hybrid_clock.cc:648] HybridClock initialized: now 1786385507365228 us; error 42 us; skew 500 ppm
I20260810 18:11:47.367013 411 webserver.cc:460] Webserver started at http://127.23.65.126:39781/ using document root <none> and password file <none>
I20260810 18:11:47.367596 411 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:47.367671 411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:47.367877 411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:47.369921 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "2903d012293c465f860605cdc2d30742"
format_stamp: "Formatted at 2026-08-10 18:11:47 on dist-test-slave-jpj1"
server_key: "c5c0674fbe03a0c4f869178794008af6"
server_key_iv: "9240a1ef22332a3eeb3bcac65dc5378f"
server_key_version: "encryptionkey@0"
I20260810 18:11:47.370498 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "2903d012293c465f860605cdc2d30742"
format_stamp: "Formatted at 2026-08-10 18:11:47 on dist-test-slave-jpj1"
server_key: "c5c0674fbe03a0c4f869178794008af6"
server_key_iv: "9240a1ef22332a3eeb3bcac65dc5378f"
server_key_version: "encryptionkey@0"
I20260810 18:11:47.374336 411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:11:47.376863 430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:47.378106 411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:47.378226 411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "2903d012293c465f860605cdc2d30742"
format_stamp: "Formatted at 2026-08-10 18:11:47 on dist-test-slave-jpj1"
server_key: "c5c0674fbe03a0c4f869178794008af6"
server_key_iv: "9240a1ef22332a3eeb3bcac65dc5378f"
server_key_version: "encryptionkey@0"
I20260810 18:11:47.378340 411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:47.396765 411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:47.397493 411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:47.397673 411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:47.405033 411 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:46601
I20260810 18:11:47.405076 520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:46601 every 8 connection(s)
I20260810 18:11:47.406240 411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:47.406453 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 411
I20260810 18:11:47.406572 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:47.406921 23813 external_mini_cluster.cc:1428] Setting key efea4d6594298aeed2433dadbe2aa0dc
I20260810 18:11:47.410001 524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:47.416483 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:47.408400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38778 (local address 127.23.65.126:46601)
0810 18:11:47.409074 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:47.409091 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:47.409129 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:11:47.409800 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:47.409819 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:47.410261 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:47.410562 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:47.410925 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.411972 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.413182 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.413656 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.413777 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:47.414697 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:47.414702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:47.414717 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:47.415117 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:47.415173 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:47.415192 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:47.415266 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:47.415427 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 18:11:47.415741 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":140,"threads_started":1}
I20260810 18:11:47.416859 524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742: Bootstrap starting.
I20260810 18:11:47.419366 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:47.419390 524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:47.420151 524 log.cc:824] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:47.422201 524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742: No bootstrap required, opened a new log
I20260810 18:11:47.426112 524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903d012293c465f860605cdc2d30742" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46601 } }
I20260810 18:11:47.426367 524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:47.426407 524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2903d012293c465f860605cdc2d30742, State: Initialized, Role: FOLLOWER
I20260810 18:11:47.426846 524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903d012293c465f860605cdc2d30742" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46601 } }
I20260810 18:11:47.426978 524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:47.427039 524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:47.427131 524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:47.428150 524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903d012293c465f860605cdc2d30742" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46601 } }
I20260810 18:11:47.428495 524 leader_election.cc:302] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2903d012293c465f860605cdc2d30742; no voters:
I20260810 18:11:47.428817 524 leader_election.cc:288] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:47.428949 533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:47.429224 533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [term 1 LEADER]: Becoming Leader. State: Replica: 2903d012293c465f860605cdc2d30742, State: Running, Role: LEADER
I20260810 18:11:47.429529 533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903d012293c465f860605cdc2d30742" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46601 } }
I20260810 18:11:47.430069 524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:47.431452 536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2903d012293c465f860605cdc2d30742. Latest consensus state: current_term: 1 leader_uuid: "2903d012293c465f860605cdc2d30742" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903d012293c465f860605cdc2d30742" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46601 } } }
I20260810 18:11:47.431615 536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:47.431680 534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2903d012293c465f860605cdc2d30742" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903d012293c465f860605cdc2d30742" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46601 } } }
I20260810 18:11:47.431769 534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742 [sys.catalog]: This master's current role is: LEADER
W20260810 18:11:47.434965 556 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:47.435057 556 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:47.435130 551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:47.435915 551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:47.442094 551 catalog_manager.cc:1347] Generated new cluster ID: edd258bd75dc436b80d9d4595a44bad9
I20260810 18:11:47.442191 551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:47.458662 551 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:47.459745 551 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:47.466837 551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2903d012293c465f860605cdc2d30742: Generated new TSK 0
I20260810 18:11:47.467519 551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:11:47.963178 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:47.478417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.23.65.126:46601)
0810 18:11:47.478543 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:47.478547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:47.478702 (+ 155us) server_negotiation.cc:407] Connection header received
0810 18:11:47.478848 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:47.478850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:47.478899 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:47.478990 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:11:47.479224 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.479801 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.481010 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.481214 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.481281 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:47.481636 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:11:47.962777 (+481141us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 18:11:47.963043 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38794: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 18:11:47.963256 529 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38794: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 18:11:47.963379 567 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:47.478270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46601 (local address 127.0.0.1:38794)
0810 18:11:47.478586 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:47.478601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:47.478722 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:47.479003 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:47.479006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:47.479014 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:11:47.479134 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:47.479141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.479908 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.479912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:47.480909 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:47.480920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:47.481311 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:47.481315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:47.481427 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:47.481463 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:11:47.963009 (+481546us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 18:11:47.963201 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.65.126:46601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":124,"threads_started":1}
I20260810 18:11:47.964094 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 411
2026-08-10T18:11:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (769 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:48 dist-test-slave-jpj1 krb5kdc[583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:48 dist-test-slave-jpj1 krb5kdc[583](info): set up 2 sockets
Aug 10 18:11:48 dist-test-slave-jpj1 krb5kdc[583](info): commencing operation
krb5kdc: starting...
W20260810 18:11:49.382926 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:49 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:11:49.507689 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:45111
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:49.619807 598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:49.620078 598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:49.620119 598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:49.624454 598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:49.624508 598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:49.624528 598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:49.624558 598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:49.624585 598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:49.629977 598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:49.631040 598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:49.632072 598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:49.640267 604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:49.640341 607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:49.640336 603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:49.640524 598 server_base.cc:1034] running on GCE node
I20260810 18:11:49.641182 598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:49.642202 598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:49.643363 598 hybrid_clock.cc:648] HybridClock initialized: now 1786385509643349 us; error 28 us; skew 500 ppm
Aug 10 18:11:49 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385509, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:49.649515 598 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:11:49.650427 598 webserver.cc:460] Webserver started at http://127.23.65.126:36917/ using document root <none> and password file <none>
I20260810 18:11:49.650967 598 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:49.651029 598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:49.651206 598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:49.653313 598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "4b6b33dbb3104eefafb9efb143223600"
format_stamp: "Formatted at 2026-08-10 18:11:49 on dist-test-slave-jpj1"
server_key: "bf4c2937f47f8da52ddf8b4d0cf5336b"
server_key_iv: "b9fbdf224945a49c836c4cc4682bde79"
server_key_version: "encryptionkey@0"
I20260810 18:11:49.653832 598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "4b6b33dbb3104eefafb9efb143223600"
format_stamp: "Formatted at 2026-08-10 18:11:49 on dist-test-slave-jpj1"
server_key: "bf4c2937f47f8da52ddf8b4d0cf5336b"
server_key_iv: "b9fbdf224945a49c836c4cc4682bde79"
server_key_version: "encryptionkey@0"
I20260810 18:11:49.657562 598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:11:49.660034 616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:49.661268 598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:49.661373 598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "4b6b33dbb3104eefafb9efb143223600"
format_stamp: "Formatted at 2026-08-10 18:11:49 on dist-test-slave-jpj1"
server_key: "bf4c2937f47f8da52ddf8b4d0cf5336b"
server_key_iv: "b9fbdf224945a49c836c4cc4682bde79"
server_key_version: "encryptionkey@0"
I20260810 18:11:49.661461 598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:49.676605 598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:49.679664 598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:49.679824 598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:49.686590 598 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:45111
I20260810 18:11:49.686618 701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:45111 every 8 connection(s)
I20260810 18:11:49.687494 598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:49.690271 702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:49.694098 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 598
I20260810 18:11:49.694237 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:49.694536 23813 external_mini_cluster.cc:1428] Setting key 9566031dde55a78f07f5a16726df1941
I20260810 18:11:49.697222 702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600: Bootstrap starting.
I20260810 18:11:49.699474 702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:49.700244 702 log.cc:824] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:49.702163 702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600: No bootstrap required, opened a new log
Aug 10 18:11:49 dist-test-slave-jpj1 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:49.705135 702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6b33dbb3104eefafb9efb143223600" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45111 } }
I20260810 18:11:49.705375 702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:49.705435 702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6b33dbb3104eefafb9efb143223600, State: Initialized, Role: FOLLOWER
I20260810 18:11:49.705863 702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6b33dbb3104eefafb9efb143223600" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45111 } }
I20260810 18:11:49.705999 702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:49.706064 702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:49.706166 702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:49.707003 702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6b33dbb3104eefafb9efb143223600" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45111 } }
I20260810 18:11:49.707329 702 leader_election.cc:302] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b6b33dbb3104eefafb9efb143223600; no voters:
I20260810 18:11:49.707698 702 leader_election.cc:288] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:49.707757 707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:49.707995 707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [term 1 LEADER]: Becoming Leader. State: Replica: 4b6b33dbb3104eefafb9efb143223600, State: Running, Role: LEADER
I20260810 18:11:49.708312 707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6b33dbb3104eefafb9efb143223600" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45111 } }
I20260810 18:11:49.708760 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:49.695967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.23.65.126:45111)
0810 18:11:49.696368 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:49.696384 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:49.696419 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:11:49.697080 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:49.697098 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:49.697463 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:49.697784 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:49.698564 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:49.699592 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:49.700445 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:49.701085 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:49.701231 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:49.703611 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:49.703617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:49.703629 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:49.703656 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:49.705722 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:49.706150 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:49.706156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:49.706160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:49.706284 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:49.706548 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:49.706551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:49.706553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:49.706954 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:49.707142 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:49.707415 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 18:11:49.707761 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260810 18:11:49.708936 702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:49.709517 707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6b33dbb3104eefafb9efb143223600. Latest consensus state: current_term: 1 leader_uuid: "4b6b33dbb3104eefafb9efb143223600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6b33dbb3104eefafb9efb143223600" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45111 } } }
I20260810 18:11:49.709645 707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:49.709981 708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b6b33dbb3104eefafb9efb143223600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6b33dbb3104eefafb9efb143223600" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45111 } } }
I20260810 18:11:49.710078 708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:49.710187 715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:49.713323 715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:49.719038 715 catalog_manager.cc:1347] Generated new cluster ID: 370bd4de0b034ef08eb566c74cdd5169
I20260810 18:11:49.719098 715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:49.740491 715 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:49.741505 715 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:49.746929 715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b6b33dbb3104eefafb9efb143223600: Generated new TSK 0
I20260810 18:11:49.747633 715 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:11:49.784581 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45111
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:49.921299 746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:49.921540 746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:49.921581 746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:49.925448 746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:49.925500 746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:49.925568 746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:49.930611 746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:49.931696 746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:49.932729 746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:49.942166 753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:49.942137 756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:49.942446 746 server_base.cc:1034] running on GCE node
W20260810 18:11:49.942195 754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:49.943027 746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:49.943673 746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:49.944841 746 hybrid_clock.cc:648] HybridClock initialized: now 1786385509944825 us; error 29 us; skew 500 ppm
Aug 10 18:11:49 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385509, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:49.947480 746 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:11:49.948423 746 webserver.cc:460] Webserver started at http://127.23.65.65:37325/ using document root <none> and password file <none>
I20260810 18:11:49.948967 746 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:49.949038 746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:49.949254 746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:49.951295 746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "54a6d543ba564ffc8ad5125977d2fffc"
format_stamp: "Formatted at 2026-08-10 18:11:49 on dist-test-slave-jpj1"
server_key: "559c0a17ad86b17dbddda69ca78d2a8a"
server_key_iv: "8245088919cc618594fb4a727f79311d"
server_key_version: "encryptionkey@0"
I20260810 18:11:49.951879 746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "54a6d543ba564ffc8ad5125977d2fffc"
format_stamp: "Formatted at 2026-08-10 18:11:49 on dist-test-slave-jpj1"
server_key: "559c0a17ad86b17dbddda69ca78d2a8a"
server_key_iv: "8245088919cc618594fb4a727f79311d"
server_key_version: "encryptionkey@0"
I20260810 18:11:49.955610 746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:49.957911 764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:49.959036 746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:49.959143 746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "54a6d543ba564ffc8ad5125977d2fffc"
format_stamp: "Formatted at 2026-08-10 18:11:49 on dist-test-slave-jpj1"
server_key: "559c0a17ad86b17dbddda69ca78d2a8a"
server_key_iv: "8245088919cc618594fb4a727f79311d"
server_key_version: "encryptionkey@0"
I20260810 18:11:49.959235 746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:49.977560 746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:49.980975 746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:49.981176 746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:49.981820 746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:49.982676 746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:49.982726 746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:49.982792 746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:49.982820 746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:49.991390 746 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:37833
I20260810 18:11:49.991423 934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:37833 every 8 connection(s)
I20260810 18:11:49.992321 746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
Aug 10 18:11:49 dist-test-slave-jpj1 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385509, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:50.001902 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 746
I20260810 18:11:50.002002 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:50.002259 23813 external_mini_cluster.cc:1428] Setting key 7fb6203d87ac9b5797f78cb68da700a0
I20260810 18:11:50.005738 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:49.994072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:33607 (local address 127.23.65.126:45111)
0810 18:11:49.994201 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:49.994205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:49.995015 (+ 810us) server_negotiation.cc:407] Connection header received
0810 18:11:49.996447 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:49.996450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:49.996511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:49.996583 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:49.997757 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:49.998295 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:49.999394 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:49.999672 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:49.999758 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:50.002117 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:50.002120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:50.002122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:50.002147 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:50.003702 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.004280 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.004283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.004285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.004366 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.004714 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.004716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.004717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.004894 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:50.004962 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:50.005516 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 18:11:50.005617 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:11:50.006665 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:49.994427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:45111 (local address 127.23.65.65:33607)
0810 18:11:49.994867 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:49.994911 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:49.996226 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:49.996765 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:49.996772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:49.997127 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:49.997609 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:49.997622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:49.998415 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:49.998418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:49.999267 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:49.999287 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:49.999821 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:49.999825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:49.999992 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:50.000518 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:50.000535 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:50.001953 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:50.003822 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:50.003829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:50.003843 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:50.004185 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:50.004452 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:50.004454 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:50.004456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:50.004631 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:50.004979 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:50.004985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:50.005281 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 18:11:50.005479 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 18:11:50.005820 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
I20260810 18:11:50.007884 935 heartbeater.cc:347] Connected to a master server at 127.23.65.126:45111
I20260810 18:11:50.008136 935 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:50.008718 935 heartbeater.cc:511] Master 127.23.65.126:45111 requested a full tablet report, sending...
I20260810 18:11:50.010412 645 ts_manager.cc:194] Registered new tserver with Master: 54a6d543ba564ffc8ad5125977d2fffc (127.23.65.65:37833)
I20260810 18:11:50.011616 645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:33607
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:11:50.032251 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45111
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:50.145623 942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:50.145884 942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:50.145929 942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:50.150010 942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:50.150065 942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:50.150149 942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:50.155242 942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:50.156270 942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:50.157245 942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:50.167048 951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:50.166994 953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:50.167001 957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:50.167047 942 server_base.cc:1034] running on GCE node
I20260810 18:11:50.167688 942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:50.168322 942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:50.169476 942 hybrid_clock.cc:648] HybridClock initialized: now 1786385510169460 us; error 27 us; skew 500 ppm
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:50.172220 942 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:11:50.173142 942 webserver.cc:460] Webserver started at http://127.23.65.66:43793/ using document root <none> and password file <none>
I20260810 18:11:50.173719 942 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:50.173794 942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:50.174011 942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:50.176211 942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "76b648b68cfb4e28b6b64a4e1693e210"
format_stamp: "Formatted at 2026-08-10 18:11:50 on dist-test-slave-jpj1"
server_key: "e8648ec78882370f298cdd367b31c4d3"
server_key_iv: "1bbc23830bf2579bdab15023f11baf43"
server_key_version: "encryptionkey@0"
I20260810 18:11:50.176781 942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "76b648b68cfb4e28b6b64a4e1693e210"
format_stamp: "Formatted at 2026-08-10 18:11:50 on dist-test-slave-jpj1"
server_key: "e8648ec78882370f298cdd367b31c4d3"
server_key_iv: "1bbc23830bf2579bdab15023f11baf43"
server_key_version: "encryptionkey@0"
I20260810 18:11:50.180644 942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:11:50.183041 968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:50.184297 942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:50.184399 942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "76b648b68cfb4e28b6b64a4e1693e210"
format_stamp: "Formatted at 2026-08-10 18:11:50 on dist-test-slave-jpj1"
server_key: "e8648ec78882370f298cdd367b31c4d3"
server_key_iv: "1bbc23830bf2579bdab15023f11baf43"
server_key_version: "encryptionkey@0"
I20260810 18:11:50.184500 942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:50.228668 942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:50.231612 942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:50.231781 942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:50.232359 942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:50.233196 942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:50.233247 942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:50.233287 942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:50.233307 942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:50.241802 942 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:38575
I20260810 18:11:50.241839 1163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:38575 every 8 connection(s)
I20260810 18:11:50.242709 942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:50.248952 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 942
I20260810 18:11:50.249063 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:50.249336 23813 external_mini_cluster.cc:1428] Setting key c24ea4eda2a81d2503a6f71c511beef9
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:50.257108 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:50.244568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:57367 (local address 127.23.65.126:45111)
0810 18:11:50.244696 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:50.244700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:50.245547 (+ 847us) server_negotiation.cc:407] Connection header received
0810 18:11:50.247022 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:50.247026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:50.247090 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:50.247161 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:50.248422 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.248970 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.250173 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.250428 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.250531 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:50.253357 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:50.253360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:50.253362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:50.253391 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:50.254864 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.255379 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.255383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.255385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.255481 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.255864 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.255867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.255868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.256123 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:50.256232 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:50.256770 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 18:11:50.256922 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":47}
I20260810 18:11:50.257872 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:50.244948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:45111 (local address 127.23.65.66:57367)
0810 18:11:50.245393 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:50.245438 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:50.246788 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:50.247348 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:50.247355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:50.247786 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:50.248278 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:50.248292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.249096 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.249099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:50.250065 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:50.250077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.250585 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.250590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:50.250804 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:50.251370 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:50.251387 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:50.253146 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:50.254994 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:50.254999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:50.255013 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:50.255278 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:50.255601 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:50.255603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:50.255605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:50.255748 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:50.256248 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:50.256256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:50.256524 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 18:11:50.256690 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 18:11:50.256971 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260810 18:11:50.259061 1165 heartbeater.cc:347] Connected to a master server at 127.23.65.126:45111
I20260810 18:11:50.259303 1165 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:50.259855 1165 heartbeater.cc:511] Master 127.23.65.126:45111 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:11:50.260932 645 ts_manager.cc:194] Registered new tserver with Master: 76b648b68cfb4e28b6b64a4e1693e210 (127.23.65.66:38575)
I20260810 18:11:50.261488 645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:57367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:11:50.277958 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45111
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:50.390403 1175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:50.390673 1175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:50.390728 1175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:50.394712 1175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:50.394763 1175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:50.394832 1175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:50.399887 1175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:50.400837 1175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:50.401834 1175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:50.411193 1186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:50.411413 1175 server_base.cc:1034] running on GCE node
W20260810 18:11:50.411201 1184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:50.411201 1183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:50.411978 1175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:50.412636 1175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:50.413802 1175 hybrid_clock.cc:648] HybridClock initialized: now 1786385510413780 us; error 36 us; skew 500 ppm
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:50.416473 1175 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:11:50.417305 1175 webserver.cc:460] Webserver started at http://127.23.65.67:40807/ using document root <none> and password file <none>
I20260810 18:11:50.417824 1175 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:50.417881 1175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:50.418052 1175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:50.420096 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "a5a98e3365b14dd0a435b312cd0a6869"
format_stamp: "Formatted at 2026-08-10 18:11:50 on dist-test-slave-jpj1"
server_key: "d416336ed23be35a45f9f41200698326"
server_key_iv: "a1e508f56815f4ce05aac5125f79bcb4"
server_key_version: "encryptionkey@0"
I20260810 18:11:50.420615 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "a5a98e3365b14dd0a435b312cd0a6869"
format_stamp: "Formatted at 2026-08-10 18:11:50 on dist-test-slave-jpj1"
server_key: "d416336ed23be35a45f9f41200698326"
server_key_iv: "a1e508f56815f4ce05aac5125f79bcb4"
server_key_version: "encryptionkey@0"
I20260810 18:11:50.424304 1175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:11:50.426596 1195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:50.427809 1175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:50.427917 1175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "a5a98e3365b14dd0a435b312cd0a6869"
format_stamp: "Formatted at 2026-08-10 18:11:50 on dist-test-slave-jpj1"
server_key: "d416336ed23be35a45f9f41200698326"
server_key_iv: "a1e508f56815f4ce05aac5125f79bcb4"
server_key_version: "encryptionkey@0"
I20260810 18:11:50.428009 1175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:50.454811 1175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:50.458021 1175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:50.458204 1175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:50.458864 1175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:50.459776 1175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:50.459829 1175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:50.459887 1175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:50.459911 1175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:50.468645 1175 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:39439
I20260810 18:11:50.468716 1371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:39439 every 8 connection(s)
I20260810 18:11:50.469530 1175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:50.474933 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 1175
I20260810 18:11:50.475044 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:50.475307 23813 external_mini_cluster.cc:1428] Setting key fe3c1944f811c9706fd3de382a43a90c
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:11:50.482754 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:50.471259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:39777 (local address 127.23.65.126:45111)
0810 18:11:50.471403 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:50.471407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:50.472240 (+ 833us) server_negotiation.cc:407] Connection header received
0810 18:11:50.473611 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:50.473614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:50.473673 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:50.473748 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:50.474930 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.475568 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.477020 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.477329 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.477423 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:50.479939 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:50.479942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:50.479944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:50.479970 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:50.481116 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.481557 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.481559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.481561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.481636 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.481904 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.481906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.481907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.482070 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:50.482140 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:50.482543 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 18:11:50.482636 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:11:50.483626 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:50.471666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:45111 (local address 127.23.65.67:39777)
0810 18:11:50.472094 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:50.472137 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:50.473399 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:50.473943 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:50.473950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:50.474306 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:50.474775 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:50.474788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.475694 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.475697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:50.476904 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:50.476915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.477489 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.477493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:50.477732 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:50.478338 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:50.478354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:50.479800 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:50.481213 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:50.481218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:50.481228 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:50.481482 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:50.481699 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:50.481701 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:50.481702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:50.481839 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:50.482138 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:50.482142 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:50.482365 (+ 223us) client_negotiation.cc:769] Sending connection context
0810 18:11:50.482511 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 18:11:50.482747 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260810 18:11:50.484817 1374 heartbeater.cc:347] Connected to a master server at 127.23.65.126:45111
I20260810 18:11:50.485047 1374 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:50.485524 1374 heartbeater.cc:511] Master 127.23.65.126:45111 requested a full tablet report, sending...
I20260810 18:11:50.486501 645 ts_manager.cc:194] Registered new tserver with Master: a5a98e3365b14dd0a435b312cd0a6869 (127.23.65.67:39439)
I20260810 18:11:50.487038 645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:39777
I20260810 18:11:50.489020 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/test-admin.keytab.
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:50.505913 23813 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 18:11:50.508630 1385 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:50.507237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.23.65.65:37833)
0810 18:11:50.507563 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:50.507571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:50.507597 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:11:50.507650 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:50.507657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:50.507844 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:50.507968 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:50.508458 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36644: BlockingRecv error: recv got EOF from 127.0.0.1:36644 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
W20260810 18:11:50.508961 23813 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:50.509903 23813 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:50.510850 23813 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:50.519752 1385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:50.511703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.23.65.65:37833)
0810 18:11:50.511817 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:50.511821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:50.511888 (+ 67us) server_negotiation.cc:407] Connection header received
0810 18:11:50.512017 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:50.512020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:50.512064 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:50.512126 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:50.512684 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.513428 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.514242 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.514614 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.514716 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:50.516725 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:50.516734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:50.516746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:50.516770 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:50.518076 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.518479 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.518484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.518485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.518563 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.518808 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.518812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.518813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.519049 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:50.519241 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:50.519455 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 18:11:50.519628 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 18:11:50.522997 1385 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:50.522254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.23.65.65:37833)
0810 18:11:50.522376 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:50.522379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:50.522420 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:11:50.522499 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:50.522501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:50.522536 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:50.522594 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:50.522899 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36654: BlockingRecv error: recv got EOF from 127.0.0.1:36654 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 18:11:50.523080 23813 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:50.524001 23813 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:50.524828 23813 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:50.532403 1385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:50.525519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.23.65.65:37833)
0810 18:11:50.525637 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:50.525640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:50.525689 (+ 49us) server_negotiation.cc:407] Connection header received
0810 18:11:50.525787 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:50.525789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:50.525821 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:50.525873 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:50.526267 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.526732 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.527403 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.527673 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.527742 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:50.529558 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:50.529561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:50.529563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:50.529593 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:50.530793 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.531204 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.531206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.531207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.531302 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.531608 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.531611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.531612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.531781 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:50.531897 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:50.532129 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 18:11:50.532227 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 18:11:50.535051 1385 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:50.534128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.23.65.65:37833)
0810 18:11:50.534245 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:50.534249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:50.534322 (+ 73us) server_negotiation.cc:407] Connection header received
0810 18:11:50.534460 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:50.534464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:50.534502 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:50.534562 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:50.534951 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36666: BlockingRecv error: recv got EOF from 127.0.0.1:36666 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 18:11:50.535125 23813 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385478759079-23813-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:50.536141 23813 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:50.536912 23813 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:11:50.544252 1385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:50.537648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.23.65.65:37833)
0810 18:11:50.537763 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:50.537768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:50.537856 (+ 88us) server_negotiation.cc:407] Connection header received
0810 18:11:50.537933 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:50.537935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:50.537969 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:50.538030 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:50.538426 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.538908 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.539678 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:50.539893 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:50.539969 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:50.541920 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:50.541923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:50.541925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:50.541947 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:50.543005 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.543336 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.543338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.543339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.543400 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:50.543617 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:50.543619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:50.543620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:50.543775 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:50.543896 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:50.544022 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 18:11:50.544107 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:11:50.545143 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 746
I20260810 18:11:50.559281 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 942
I20260810 18:11:50.583397 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 1175
I20260810 18:11:50.625010 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 598
2026-08-10T18:11:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2404 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[1407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[1407](info): set up 2 sockets
Aug 10 18:11:50 dist-test-slave-jpj1 krb5kdc[1407](info): commencing operation
krb5kdc: starting...
W20260810 18:11:51.780727 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:51 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.126.keytab.
Entry for principal oryx/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.126.keytab.
I20260810 18:11:51.907986 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46139
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.126.keytab
--principal=oryx/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:52.021056 1424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:52.021369 1424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:52.021410 1424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:52.025482 1424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:52.025552 1424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:52.025570 1424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:52.025591 1424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:52.025606 1424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:52.031108 1424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.126.keytab
--principal=oryx/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:52.032344 1424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:52.033414 1424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:11:52.042284 1424 server_base.cc:1034] running on GCE node
W20260810 18:11:52.042217 1431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:52.042241 1430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:52.042203 1434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:52.043128 1424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:52.044360 1424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:52.045538 1424 hybrid_clock.cc:648] HybridClock initialized: now 1786385512045516 us; error 32 us; skew 500 ppm
Aug 10 18:11:52 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385512, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:52.048692 1424 init.cc:377] Logged in from keytab as oryx/127.23.65.126@KRBTEST.COM (short username oryx)
I20260810 18:11:52.049649 1424 webserver.cc:460] Webserver started at http://127.23.65.126:42361/ using document root <none> and password file <none>
I20260810 18:11:52.050205 1424 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:52.050276 1424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:52.050484 1424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:52.052660 1424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "be8f8f35d27fa025efdb4b6028e7eea2"
server_key_iv: "07fd56cba93fdf0bfa91aebcd9352ef2"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.053229 1424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "be8f8f35d27fa025efdb4b6028e7eea2"
server_key_iv: "07fd56cba93fdf0bfa91aebcd9352ef2"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.057031 1424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:11:52.059430 1445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.060699 1424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:52.060845 1424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "be8f8f35d27fa025efdb4b6028e7eea2"
server_key_iv: "07fd56cba93fdf0bfa91aebcd9352ef2"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.060958 1424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:52.081521 1424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:52.084589 1424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:52.084795 1424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:52.092037 1424 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:46139
I20260810 18:11:52.092321 1522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:46139 every 8 connection(s)
I20260810 18:11:52.093089 1424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:52.094888 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 1424
I20260810 18:11:52.095037 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:52.095362 23813 external_mini_cluster.cc:1428] Setting key 94a5a51ff8558a0fc5f1614a02cdc488
I20260810 18:11:52.096396 1523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:52.103099 1523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: Bootstrap starting.
Aug 10 18:11:52 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:52.105844 1523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:52.106774 1523 log.cc:824] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:52.109458 1523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: No bootstrap required, opened a new log
I20260810 18:11:52.110656 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:52.096871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.23.65.126:46139)
0810 18:11:52.097290 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:52.097306 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:52.097345 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:11:52.098018 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:52.098035 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:52.098367 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:52.098656 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:52.099596 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.100658 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.101801 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.102280 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.102396 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:52.105152 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:52.105159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:52.105176 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:52.105211 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:52.107109 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:52.107764 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:52.107780 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:52.107787 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:52.107948 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:52.108238 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:52.108241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:52.108243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:52.108795 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:52.109000 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:52.109281 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 18:11:52.109631 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
I20260810 18:11:52.113793 1523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } }
I20260810 18:11:52.114032 1523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:52.114065 1523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b729eb5df9e496cbbfeaa1fc28ed39e, State: Initialized, Role: FOLLOWER
I20260810 18:11:52.114454 1523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } }
I20260810 18:11:52.114583 1523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:52.114648 1523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:52.114748 1523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:52.115784 1523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } }
I20260810 18:11:52.116110 1523 leader_election.cc:302] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b729eb5df9e496cbbfeaa1fc28ed39e; no voters:
I20260810 18:11:52.116425 1523 leader_election.cc:288] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:52.116555 1529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:52.116778 1529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 1 LEADER]: Becoming Leader. State: Replica: 2b729eb5df9e496cbbfeaa1fc28ed39e, State: Running, Role: LEADER
I20260810 18:11:52.117111 1529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } }
I20260810 18:11:52.117652 1523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:52.118819 1533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b729eb5df9e496cbbfeaa1fc28ed39e. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } }
I20260810 18:11:52.118935 1533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:52.119259 1531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } }
I20260810 18:11:52.119391 1531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:52.119362 1544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:52.122401 1544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:52.128137 1544 catalog_manager.cc:1347] Generated new cluster ID: 832069d7906941fa973f3b2ee8c3eb56
I20260810 18:11:52.128211 1544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:52.159610 1544 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:52.160970 1544 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:52.180801 1544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: Generated new TSK 0
I20260810 18:11:52.181839 1544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab.
Entry for principal oryx/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab.
I20260810 18:11:52.226946 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46139
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab
--principal=oryx/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:52.345834 1565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:52.346112 1565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:52.346179 1565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:52.350246 1565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:52.350322 1565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:52.350404 1565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:52.355939 1565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab
--principal=oryx/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:52.357108 1565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:52.358219 1565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:52.368245 1577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:52.368227 1575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:52.368287 1565 server_base.cc:1034] running on GCE node
W20260810 18:11:52.368227 1579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:52.368999 1565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:52.369668 1565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:52.370847 1565 hybrid_clock.cc:648] HybridClock initialized: now 1786385512370830 us; error 36 us; skew 500 ppm
Aug 10 18:11:52 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385512, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:52.373560 1565 init.cc:377] Logged in from keytab as oryx/127.23.65.65@KRBTEST.COM (short username oryx)
I20260810 18:11:52.374491 1565 webserver.cc:460] Webserver started at http://127.23.65.65:45415/ using document root <none> and password file <none>
I20260810 18:11:52.375053 1565 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:52.375123 1565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:52.375324 1565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:52.377403 1565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "e92666cfe2574cff8a51c370ad089327"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "dcafd4c3e777377e69ef20aff1f574b0"
server_key_iv: "e74bf55a0ef009f660be56f551252ff0"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.377996 1565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "e92666cfe2574cff8a51c370ad089327"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "dcafd4c3e777377e69ef20aff1f574b0"
server_key_iv: "e74bf55a0ef009f660be56f551252ff0"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.381711 1565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:11:52.384119 1592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.385304 1565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:52.385427 1565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "e92666cfe2574cff8a51c370ad089327"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "dcafd4c3e777377e69ef20aff1f574b0"
server_key_iv: "e74bf55a0ef009f660be56f551252ff0"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.385524 1565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:52.405819 1565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:52.410887 1565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:52.412750 1565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:52.413554 1565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:52.414641 1565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:52.414705 1565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.414767 1565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:52.414793 1565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.430125 1565 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:34559
I20260810 18:11:52.430174 1765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:34559 every 8 connection(s)
I20260810 18:11:52.431108 1565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:52.434749 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 1565
I20260810 18:11:52.434870 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:52.435195 23813 external_mini_cluster.cc:1428] Setting key f685fee9cd5d1d5443c50a85dbdf5e9a
Aug 10 18:11:52 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385512, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:11:52.446900 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:52.433345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:58851 (local address 127.23.65.126:46139)
0810 18:11:52.433486 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:52.433491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:52.434582 (+ 1091us) server_negotiation.cc:407] Connection header received
0810 18:11:52.436765 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:52.436769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:52.436824 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:52.436898 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:52.438109 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.438845 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.440430 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.440764 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.440868 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:52.443376 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:52.443379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:52.443381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:52.443404 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:52.444792 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:52.445317 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:52.445320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:52.445322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:52.445401 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:52.445759 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:52.445762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:52.445764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:52.445945 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:52.446035 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:52.446590 (+ 555us) server_negotiation.cc:299] Negotiation successful
0810 18:11:52.446726 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:11:52.447844 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:52.433810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.65:58851)
0810 18:11:52.434373 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:52.434438 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:52.436489 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:52.437077 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:52.437084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:52.437469 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:52.437965 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:52.437980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.439012 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.439016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:52.440305 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:52.440317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.440926 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.440930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:52.441088 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:52.441753 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:52.441775 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:52.443206 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:52.444921 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:52.444927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:52.444942 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:52.445218 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:52.445488 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:52.445491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:52.445493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:52.445667 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:52.446049 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:52.446057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:52.446313 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 18:11:52.446522 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 18:11:52.446875 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.66@KRBTEST.COM" created.
I20260810 18:11:52.449148 1767 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46139
I20260810 18:11:52.449388 1767 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:52.449893 1767 heartbeater.cc:511] Master 127.23.65.126:46139 requested a full tablet report, sending...
I20260810 18:11:52.451454 1466 ts_manager.cc:194] Registered new tserver with Master: e92666cfe2574cff8a51c370ad089327 (127.23.65.65:34559)
I20260810 18:11:52.452683 1466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:58851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab.
Entry for principal oryx/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab.
I20260810 18:11:52.467439 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46139
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab
--principal=oryx/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:52.582434 1777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:52.582706 1777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:52.582768 1777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:52.586778 1777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:52.586841 1777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:52.586923 1777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:52.592253 1777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab
--principal=oryx/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:52.593418 1777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:52.594471 1777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:52.603955 1785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:52.603968 1786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:52.604084 1788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:52.604285 1777 server_base.cc:1034] running on GCE node
I20260810 18:11:52.604789 1777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:52.605419 1777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:52.606575 1777 hybrid_clock.cc:648] HybridClock initialized: now 1786385512606550 us; error 44 us; skew 500 ppm
Aug 10 18:11:52 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385512, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:52.609337 1777 init.cc:377] Logged in from keytab as oryx/127.23.65.66@KRBTEST.COM (short username oryx)
I20260810 18:11:52.610306 1777 webserver.cc:460] Webserver started at http://127.23.65.66:46623/ using document root <none> and password file <none>
I20260810 18:11:52.610963 1777 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:52.611050 1777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:52.611240 1777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:52.613310 1777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "70d63c94c94d4c77995cbf795453d93e"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "1bdf49d73afbecc95effea584c565f76"
server_key_iv: "0574d82074bcc60f0aaff8d64bf76b9c"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.613845 1777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "70d63c94c94d4c77995cbf795453d93e"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "1bdf49d73afbecc95effea584c565f76"
server_key_iv: "0574d82074bcc60f0aaff8d64bf76b9c"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.617733 1777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:11:52.620360 1805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.621563 1777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:52.621662 1777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "70d63c94c94d4c77995cbf795453d93e"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "1bdf49d73afbecc95effea584c565f76"
server_key_iv: "0574d82074bcc60f0aaff8d64bf76b9c"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.621783 1777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:52.633719 1777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:52.636884 1777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:52.637049 1777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:52.637627 1777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:52.638465 1777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:52.638515 1777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.638556 1777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:52.638576 1777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.646967 1777 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:40939
I20260810 18:11:52.647019 1973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:40939 every 8 connection(s)
I20260810 18:11:52.647897 1777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:52.654740 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 1777
I20260810 18:11:52.654868 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:52.655136 23813 external_mini_cluster.cc:1428] Setting key 31f563fd10d1c6e374d5c072667c755c
Aug 10 18:11:52 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385512, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:52.662176 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:52.649712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:59569 (local address 127.23.65.126:46139)
0810 18:11:52.649847 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:52.649850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:52.650957 (+ 1107us) server_negotiation.cc:407] Connection header received
0810 18:11:52.652450 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:52.652454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:52.652514 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:52.652588 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:52.653820 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.654360 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.655605 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.655850 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.655952 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:52.658612 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:52.658615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:52.658618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:52.658645 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:52.660038 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:52.660561 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:52.660564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:52.660566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:52.660651 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:52.660987 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:52.660991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:52.660993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:52.661184 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:52.661281 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:52.661937 (+ 656us) server_negotiation.cc:299] Negotiation successful
0810 18:11:52.662046 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:11:52.663379 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:52.650228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.66:59569)
0810 18:11:52.650789 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:52.650840 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:52.652200 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:52.652778 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:52.652785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:52.653154 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:52.653660 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:52.653674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.654489 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.654492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:52.655462 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:52.655473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.655996 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.655999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:52.656161 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:52.656789 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:52.656810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:52.658445 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:52.660154 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:52.660161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:52.660178 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:52.660462 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:52.660739 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:52.660742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:52.660744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:52.660896 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:52.661313 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:52.661321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:52.661666 (+ 345us) client_negotiation.cc:769] Sending connection context
0810 18:11:52.661905 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 18:11:52.662275 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":127,"threads_started":1}
I20260810 18:11:52.664605 1974 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46139
I20260810 18:11:52.664889 1974 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:52.665548 1974 heartbeater.cc:511] Master 127.23.65.126:46139 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:11:52.666765 1466 ts_manager.cc:194] Registered new tserver with Master: 70d63c94c94d4c77995cbf795453d93e (127.23.65.66:40939)
I20260810 18:11:52.667366 1466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:59569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab.
Entry for principal oryx/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab.
I20260810 18:11:52.686149 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46139
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab
--principal=oryx/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:52.799613 1982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:52.799885 1982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:52.799933 1982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:52.804193 1982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:52.804260 1982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:52.804339 1982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:52.809831 1982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab
--principal=oryx/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:52.810940 1982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:52.811997 1982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:52.822650 1994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:52.822772 1982 server_base.cc:1034] running on GCE node
W20260810 18:11:52.822636 1989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:52.822631 1992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:52.823352 1982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:52.824036 1982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:52.825204 1982 hybrid_clock.cc:648] HybridClock initialized: now 1786385512825185 us; error 31 us; skew 500 ppm
Aug 10 18:11:52 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385512, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:52.827850 1982 init.cc:377] Logged in from keytab as oryx/127.23.65.67@KRBTEST.COM (short username oryx)
I20260810 18:11:52.828683 1982 webserver.cc:460] Webserver started at http://127.23.65.67:37785/ using document root <none> and password file <none>
I20260810 18:11:52.829218 1982 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:52.829284 1982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:52.829484 1982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:52.831511 1982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "56e697639a4d4f519a9fa399614b49b8"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "c760323f66ba2018aa9b62b416574023"
server_key_iv: "fa0acb02be65d198b1472dbbf4d6eeab"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.832192 1982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "56e697639a4d4f519a9fa399614b49b8"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "c760323f66ba2018aa9b62b416574023"
server_key_iv: "fa0acb02be65d198b1472dbbf4d6eeab"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.836140 1982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260810 18:11:52.838620 2004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.839957 1982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:52.840071 1982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "56e697639a4d4f519a9fa399614b49b8"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "c760323f66ba2018aa9b62b416574023"
server_key_iv: "fa0acb02be65d198b1472dbbf4d6eeab"
server_key_version: "encryptionkey@0"
I20260810 18:11:52.840179 1982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:52.853432 1982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:52.856325 1982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:52.856492 1982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:52.857044 1982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:52.857826 1982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:52.857873 1982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.857924 1982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:52.857954 1982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:52.866528 1982 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:40235
I20260810 18:11:52.866565 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:40235 every 8 connection(s)
I20260810 18:11:52.867377 1982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:52.873476 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 1982
I20260810 18:11:52.873656 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:52.874008 23813 external_mini_cluster.cc:1428] Setting key ed4a18154c900a3280b1489e3c7d6a09
Aug 10 18:11:52 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385512, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:52.880797 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:52.869162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:60293 (local address 127.23.65.126:46139)
0810 18:11:52.869311 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:52.869316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:52.870161 (+ 845us) server_negotiation.cc:407] Connection header received
0810 18:11:52.871489 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:52.871493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:52.871576 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:52.871656 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:52.872802 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.873336 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.874827 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.875136 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.875232 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:52.877872 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:52.877875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:52.877877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:52.877902 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:52.879036 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:52.879483 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:52.879485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:52.879486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:52.879579 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:52.879867 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:52.879869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:52.879870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:52.880043 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:52.880116 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:52.880569 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 18:11:52.880668 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:11:52.881654 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:52.869542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.67:60293)
0810 18:11:52.870003 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:52.870047 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:52.871280 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:52.871810 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:52.871817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:52.872174 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:52.872681 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:52.872697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.873468 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.873472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:52.874715 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:52.874726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:52.875289 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:52.875292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:52.875481 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:52.876163 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:52.876182 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:52.877708 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:52.879147 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:52.879152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:52.879165 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:52.879410 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:52.879662 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:52.879664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:52.879666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:52.879798 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:52.880132 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:52.880138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:52.880378 (+ 240us) client_negotiation.cc:769] Sending connection context
0810 18:11:52.880531 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 18:11:52.880790 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260810 18:11:52.882856 2177 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46139
I20260810 18:11:52.883095 2177 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:52.883636 2177 heartbeater.cc:511] Master 127.23.65.126:46139 requested a full tablet report, sending...
I20260810 18:11:52.884654 1466 ts_manager.cc:194] Registered new tserver with Master: 56e697639a4d4f519a9fa399614b49b8 (127.23.65.67:40235)
I20260810 18:11:52.885192 1466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:60293
I20260810 18:11:52.887800 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.65.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.125.keytab.
Entry for principal oryx/127.23.65.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.125.keytab.
I20260810 18:11:52.916090 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.125:38417
--webserver_interface=127.23.65.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.125.keytab
--principal=oryx/127.23.65.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46139,127.23.65.125:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:53.029475 2182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:53.029747 2182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:53.029796 2182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:53.033774 2182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:53.033857 2182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:53.033876 2182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:53.033891 2182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:53.033906 2182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:53.039366 2182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46139,127.23.65.125:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.125.keytab
--principal=oryx/127.23.65.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.125:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.65.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
W20260810 18:11:53.039633 2182 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.65.126:46139,127.23.65.125:38417'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 18:11:53.040652 2182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:53.041715 2182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:53.050118 2190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.050118 2191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.050128 2194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:53.050406 2182 server_base.cc:1034] running on GCE node
I20260810 18:11:53.051047 2182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:53.052155 2182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:53.053324 2182 hybrid_clock.cc:648] HybridClock initialized: now 1786385513053303 us; error 37 us; skew 500 ppm
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:53.056062 2182 init.cc:377] Logged in from keytab as oryx/127.23.65.125@KRBTEST.COM (short username oryx)
I20260810 18:11:53.056983 2182 webserver.cc:460] Webserver started at http://127.23.65.125:40603/ using document root <none> and password file <none>
I20260810 18:11:53.057519 2182 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:53.057585 2182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:53.057796 2182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:53.059893 2182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data/instance:
uuid: "510ce43378724dd8816f3b2dd123516f"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "fee851a3089b805c9a8c1a00531adbf7"
server_key_iv: "82405ff4fca5311419931f5db89b469d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.060457 2182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal/instance:
uuid: "510ce43378724dd8816f3b2dd123516f"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "fee851a3089b805c9a8c1a00531adbf7"
server_key_iv: "82405ff4fca5311419931f5db89b469d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.064337 2182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:11:53.066977 2206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.068327 2182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:53.068445 2182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal
uuid: "510ce43378724dd8816f3b2dd123516f"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "fee851a3089b805c9a8c1a00531adbf7"
server_key_iv: "82405ff4fca5311419931f5db89b469d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.068554 2182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:53.085757 2182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:53.088948 2182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:53.089128 2182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:53.096647 2182 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.125:38417
I20260810 18:11:53.097047 2286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.125:38417 every 8 connection(s)
I20260810 18:11:53.097615 2182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
I20260810 18:11:53.101044 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:53.103317 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 2182
I20260810 18:11:53.105721 2287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.65.125@KRBTEST.COM
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.125@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:53.117378 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.107997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.23.65.126:46139)
0810 18:11:53.108106 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.108110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.108568 (+ 458us) server_negotiation.cc:407] Connection header received
0810 18:11:53.108694 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.108698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.108742 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.108807 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.109607 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.110204 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.111249 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.111470 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.111601 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.113939 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.113942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.113945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.113970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.115496 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.115972 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.115975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.115976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.116054 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.116333 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.116335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.116336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.116606 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.116711 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.117140 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.117244 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:11:53.118283 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.104920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.23.65.125:38417)
0810 18:11:53.105326 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.105343 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.105377 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:11:53.106020 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.106036 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.106364 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.106685 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.107621 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.108988 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.109998 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.110587 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.110701 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.113382 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.113390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.113404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.113430 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.114952 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.115412 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.115416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.115421 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.115524 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.115859 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.115862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.115863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.116378 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.116549 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.116853 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.117156 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260810 18:11:53.118289 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.107979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:44462)
0810 18:11:53.108352 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.108392 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.108609 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.108878 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.108884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.109310 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.109473 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.109482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.110339 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.110342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.111134 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.111144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.111664 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.111668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.111792 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.112298 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.112320 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.113776 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.115611 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.115621 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.115628 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.115873 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.116135 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.116138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.116139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.116225 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.116713 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.116719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.116865 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.117052 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.117206 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"spinlock_wait_cycles":896,"thread_start_us":138,"threads_started":1}
I20260810 18:11:53.120046 2287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.125@KRBTEST.COM for oryx/127.23.65.125@KRBTEST.COM
I20260810 18:11:53.127647 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.120756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.0.0.1:59626)
0810 18:11:53.120866 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.120879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.120952 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.121136 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.121139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.121287 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.121386 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.121391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.122421 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.122424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.122915 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.122922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.123332 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.123335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.123425 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.123785 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.123795 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.124911 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.126282 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.126286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.126288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.126474 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.126719 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.126722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.126724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.126797 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.127126 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.127129 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.127212 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.127293 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.127437 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 18:11:53.127651 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.120803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.23.65.125:38417)
0810 18:11:53.120892 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.120896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.120912 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:11:53.121017 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.121020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.121054 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.121116 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.121492 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.122318 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.123014 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.123236 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.123319 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.125030 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.125034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.125036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.125056 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.126136 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.126562 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.126565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.126567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.126641 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.126875 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.126878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.126880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.127043 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.127126 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.127351 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.127494 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":36}
I20260810 18:11:53.130556 2287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: Bootstrap starting.
I20260810 18:11:53.132865 2287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:53.133580 2287 log.cc:824] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:53.135445 2287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: No bootstrap required, opened a new log
I20260810 18:11:53.138489 2287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } }
I20260810 18:11:53.138702 2287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:53.138734 2287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510ce43378724dd8816f3b2dd123516f, State: Initialized, Role: FOLLOWER
I20260810 18:11:53.139118 2287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } }
I20260810 18:11:53.139599 2294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } } }
I20260810 18:11:53.139731 2294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:53.140352 2287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:11:53.145117 2313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:53.145195 2313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:53.147580 2182 master_runner.cc:428] Detected that this master 510ce43378724dd8816f3b2dd123516f is joining an existing cluster
I20260810 18:11:53.147641 2182 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.65.125:38417
I20260810 18:11:53.158088 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.150054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.23.65.125:38417)
0810 18:11:53.150162 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.150166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.150322 (+ 156us) server_negotiation.cc:407] Connection header received
0810 18:11:53.150431 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.150433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.150467 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.150510 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.151040 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.151519 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.152679 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.153028 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.153141 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.154911 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.154915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.154917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.154945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.156408 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.156949 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.156953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.156955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.157048 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.157292 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.157296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.157298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.157485 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.157569 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.157820 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.157940 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":45}
I20260810 18:11:53.158102 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.149772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.0.0.1:59634)
0810 18:11:53.150248 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.150259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.150348 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.150644 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.150646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.150832 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.150949 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.150956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.151655 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.151659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.152540 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.152551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.153161 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.153164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.153249 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.154406 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.154421 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.154799 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.156506 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.156509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.156511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.156845 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.157132 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.157134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.157135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.157209 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.157568 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.157570 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.157657 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.157740 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.157890 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":103,"threads_started":1}
I20260810 18:11:53.158478 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.149775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.23.65.126:46139)
0810 18:11:53.150095 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.150100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.150114 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:53.150243 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.150247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.150307 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.150387 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.150953 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.151811 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.152844 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.153088 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.153185 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.154749 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.154753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.154756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.154785 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.156453 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.157171 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.157175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.157177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.157262 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.157546 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.157549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.157551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.157759 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.157840 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.158194 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.158319 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260810 18:11:53.159230 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.149713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:44478)
0810 18:11:53.149987 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.150002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.150106 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.150410 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.150413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.150679 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.150826 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.150834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.151925 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.151928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.152740 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.152747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.153310 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.153313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.153395 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.154025 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.154043 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.154583 (+ 540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.156551 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.156553 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.156555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.156853 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.157361 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.157364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.157366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.157450 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.157914 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.157917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.158015 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.158903 (+ 888us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.159047 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"spinlock_wait_cycles":896,"thread_start_us":108,"threads_started":1}
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385512, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.126@KRBTEST.COM for oryx/127.23.65.125@KRBTEST.COM
I20260810 18:11:53.172580 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.163891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.0.0.1:59636)
0810 18:11:53.164248 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.164302 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.164434 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.164739 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.164746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.165127 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.165261 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.165267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.166048 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.166052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.166692 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.166701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.167120 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.167123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.167246 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.167768 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.167782 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.169125 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.170613 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.170623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.170630 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.170925 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.171343 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.171347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.171349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.171447 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.171905 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.171913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.172050 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.172205 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.172411 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
I20260810 18:11:53.172952 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.163910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.23.65.125:38417)
0810 18:11:53.164009 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.164013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.164377 (+ 364us) server_negotiation.cc:407] Connection header received
0810 18:11:53.164556 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.164560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.164608 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.164671 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.165381 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.165941 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.166810 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.167020 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.167106 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.169254 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.169258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.169261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.169285 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.170512 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.171007 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.171010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.171012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.171087 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.171566 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.171570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.171571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.171797 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.171906 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.172422 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.172540 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:11:53.174500 1466 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.65.125:38417: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "510ce43378724dd8816f3b2dd123516f"
member_type: NON_VOTER
last_known_addr {
host: "127.23.65.125"
port: 38417
}
attrs {
promote: true
}
}
dest_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e"
cas_config_opid_index: -1
I20260810 18:11:53.175261 1466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } }
I20260810 18:11:53.177098 2337 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 510ce43378724dd8816f3b2dd123516f (127.23.65.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } } }
I20260810 18:11:53.177888 2253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:53.179227 2337 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.65.125:38417
I20260810 18:11:53.179236 2253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Refusing update from remote peer 2b729eb5df9e496cbbfeaa1fc28ed39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 18:11:53.179831 2182 master.cc:537] Master@127.23.65.125:38417 shutting down...
I20260810 18:11:53.180183 2339 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:11:53.182164 1449 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.23.65.124@KRBTEST.COM; defaulting to no policy
W20260810 18:11:53.183274 1449 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e -> Peer 510ce43378724dd8816f3b2dd123516f (127.23.65.125:38417): Couldn't send request to peer 510ce43378724dd8816f3b2dd123516f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:11:53.183694 2335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } } }
I20260810 18:11:53.183804 2335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:53.185097 2182 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:53.185352 2182 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:53.185459 2182 tablet_replica.cc:330] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.124@KRBTEST.COM" created.
I20260810 18:11:53.190696 2182 master.cc:552] Master@127.23.65.125:38417 shutdown complete.
I20260810 18:11:53.190739 2182 master_runner.cc:305] Clearing existing system tablet
I20260810 18:11:53.190954 2182 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:53.191004 2182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:53.192095 2182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 18:11:53.192878 2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.193084 2182 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:53.193212 2182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 18:11:53.193250 2182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal
uuid: "510ce43378724dd8816f3b2dd123516f"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "fee851a3089b805c9a8c1a00531adbf7"
server_key_iv: "82405ff4fca5311419931f5db89b469d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.194995 2182 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:53.197686 2182 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 18:11:53.197772 2182 log.cc:1196] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 18:11:53.197984 2182 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: Deleting consensus metadata
I20260810 18:11:53.198156 2182 master_runner.cc:315] Copying system tablet from 127.23.65.126:46139
I20260810 18:11:53.199482 2182 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: tablet copy: Beginning tablet copy session from remote peer at address 127.23.65.126:46139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.124.keytab.
Entry for principal oryx/127.23.65.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.124.keytab.
I20260810 18:11:53.204418 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.124:36711
--webserver_interface=127.23.65.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.124.keytab
--principal=oryx/127.23.65.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46139,127.23.65.125:38417,127.23.65.124:36711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:11:53.210515 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.200241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.23.65.126:46139)
0810 18:11:53.200397 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.200402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.200529 (+ 127us) server_negotiation.cc:407] Connection header received
0810 18:11:53.200644 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.200648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.200708 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.200793 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.201306 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.201844 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.203045 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.203264 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.203371 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.206939 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.206944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.206947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.206977 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.208669 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.209203 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.209206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.209208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.209297 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.209595 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.209598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.209600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.209833 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.209934 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.210226 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.210350 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:11:53.210539 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.200143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:44494)
0810 18:11:53.200413 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.200428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.200553 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.200828 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.200831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.201048 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.201170 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.201177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.201966 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.201970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.202922 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.202935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.203393 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.203396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.203483 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.206329 (+ 2846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.206346 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.206766 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.208803 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.208807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.208809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.209099 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.209389 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.209393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.209395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.209494 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.209933 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.209936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.210056 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.210168 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.210322 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
I20260810 18:11:53.211177 1495 tablet_copy_service.cc:140] P 2b729eb5df9e496cbbfeaa1fc28ed39e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 510ce43378724dd8816f3b2dd123516f ({username='oryx', principal='oryx/127.23.65.125@KRBTEST.COM'} at 127.0.0.1:44494)
I20260810 18:11:53.211378 1495 tablet_copy_service.cc:161] P 2b729eb5df9e496cbbfeaa1fc28ed39e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 510ce43378724dd8816f3b2dd123516f at {username='oryx', principal='oryx/127.23.65.125@KRBTEST.COM'} at 127.0.0.1:44494: session id = 510ce43378724dd8816f3b2dd123516f-00000000000000000000000000000000
I20260810 18:11:53.214098 1495 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:11:53.216234 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 18:11:53.218742 2182 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: tablet copy: Starting download of 0 data blocks...
I20260810 18:11:53.218954 2182 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: tablet copy: Starting download of 1 WAL segments...
I20260810 18:11:53.220887 2182 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:11:53.222733 1495 tablet_copy_service.cc:342] P 2b729eb5df9e496cbbfeaa1fc28ed39e: Request end of tablet copy session 510ce43378724dd8816f3b2dd123516f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.65.125@KRBTEST.COM'} at 127.0.0.1:44494
I20260810 18:11:53.222875 1495 tablet_copy_service.cc:433] P 2b729eb5df9e496cbbfeaa1fc28ed39e: ending tablet copy session 510ce43378724dd8816f3b2dd123516f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 510ce43378724dd8816f3b2dd123516f
I20260810 18:11:53.230964 2182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:53.232786 2369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.232864 2359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.233355 2360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:53.235744 2182 server_base.cc:1034] running on GCE node
I20260810 18:11:53.235936 2182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:53.236282 2182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:53.237427 2182 hybrid_clock.cc:648] HybridClock initialized: now 1786385513237415 us; error 35 us; skew 500 ppm
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:53.239049 2182 init.cc:377] Logged in from keytab as oryx/127.23.65.125@KRBTEST.COM (short username oryx)
I20260810 18:11:53.239578 2182 webserver.cc:460] Webserver started at http://127.23.65.125:43377/ using document root <none> and password file <none>
I20260810 18:11:53.239809 2182 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:53.239856 2182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:53.241194 2182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:53.242199 2376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.242478 2182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:53.242581 2182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal
uuid: "510ce43378724dd8816f3b2dd123516f"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "fee851a3089b805c9a8c1a00531adbf7"
server_key_iv: "82405ff4fca5311419931f5db89b469d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.242666 2182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:53.268759 2182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:53.271044 2182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:53.277088 2182 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.125:38417
I20260810 18:11:53.277120 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.125:38417 every 8 connection(s)
I20260810 18:11:53.277693 2182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
I20260810 18:11:53.279618 2464 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:11:53.280737 2464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: Bootstrap starting.
I20260810 18:11:53.291075 2464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:11:53.291373 2464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: Bootstrap complete.
I20260810 18:11:53.291937 2464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } }
I20260810 18:11:53.292071 2464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 510ce43378724dd8816f3b2dd123516f, State: Initialized, Role: LEARNER
I20260810 18:11:53.292165 2464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } }
I20260810 18:11:53.292490 2467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } } }
I20260810 18:11:53.292590 2464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:53.292722 2467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: This master's current role is: LEARNER
I20260810 18:11:53.295496 2490 catalog_manager.cc:1259] Loaded cluster ID: 832069d7906941fa973f3b2ee8c3eb56
I20260810 18:11:53.295656 2490 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: loading cluster ID for follower catalog manager: success
I20260810 18:11:53.297849 2490 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: acquiring CA information for follower catalog manager: success
I20260810 18:11:53.299225 2490 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 18:11:53.337898 2358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:53.338171 2358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:53.338230 2358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:53.342208 2358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:53.342260 2358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:53.342288 2358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:53.342309 2358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:53.342337 2358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:53.347615 2358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46139,127.23.65.125:38417,127.23.65.124:36711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.124.keytab
--principal=oryx/127.23.65.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.124:36711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.65.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:53.348640 2358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:53.349656 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:53.358022 2496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.358028 2498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.358038 2495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:53.358311 2358 server_base.cc:1034] running on GCE node
I20260810 18:11:53.358906 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:53.360132 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:53.361300 2358 hybrid_clock.cc:648] HybridClock initialized: now 1786385513361284 us; error 31 us; skew 500 ppm
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:53.364030 2358 init.cc:377] Logged in from keytab as oryx/127.23.65.124@KRBTEST.COM (short username oryx)
I20260810 18:11:53.364964 2358 webserver.cc:460] Webserver started at http://127.23.65.124:34509/ using document root <none> and password file <none>
I20260810 18:11:53.365522 2358 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:53.365592 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:53.365803 2358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:53.367841 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data/instance:
uuid: "32ee363451724bd89da561e507770d1c"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "1774a4f92adc949876a44c370897bd53"
server_key_iv: "5951a52d8e235041e3f31e18e89f036d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.368412 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal/instance:
uuid: "32ee363451724bd89da561e507770d1c"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "1774a4f92adc949876a44c370897bd53"
server_key_iv: "5951a52d8e235041e3f31e18e89f036d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.372087 2358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:11:53.374364 2511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.375494 2358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:53.375648 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal
uuid: "32ee363451724bd89da561e507770d1c"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "1774a4f92adc949876a44c370897bd53"
server_key_iv: "5951a52d8e235041e3f31e18e89f036d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.375738 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:53.394037 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:53.396890 2358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:53.397065 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:53.406994 2358 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.124:36711
I20260810 18:11:53.407287 2582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.124:36711 every 8 connection(s)
I20260810 18:11:53.408524 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
I20260810 18:11:53.412497 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 2358
I20260810 18:11:53.416584 2583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:53.421490 2583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.65.124@KRBTEST.COM
I20260810 18:11:53.428555 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.414235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.23.65.124:36711)
0810 18:11:53.414749 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.414776 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.414825 (+ 49us) server_negotiation.cc:407] Connection header received
0810 18:11:53.415731 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.415755 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.416195 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.416562 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.417416 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.418442 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.419679 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.420304 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.420428 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.423377 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.423382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.423395 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.423423 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.425595 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.426113 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.426117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.426122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.426238 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.426520 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.426523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.426524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.426919 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.427131 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.427414 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.427725 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":141,"threads_started":1}
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.124@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:53.432689 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.423508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:44498)
0810 18:11:53.424026 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.424084 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.424221 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.424491 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.424496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.424990 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.425154 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.425162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.426046 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.426050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.426811 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.426821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.427243 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.427246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.427378 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.427957 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.427969 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.429304 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.431130 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.431137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.431141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.431364 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.431666 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.431668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.431669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.431746 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.432145 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.432150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.432250 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.432369 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.432513 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260810 18:11:53.432727 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.423669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.23.65.126:46139)
0810 18:11:53.423817 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.423822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.424178 (+ 356us) server_negotiation.cc:407] Connection header received
0810 18:11:53.424322 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.424326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.424376 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.424442 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.425286 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.425883 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.426921 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.427145 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.427232 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.429458 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.429461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.429463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.429488 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.431027 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.431456 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.431459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.431461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.431588 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.431826 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.431829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.431831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.432056 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.432156 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.432447 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.432577 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:11:53.434038 2583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.124@KRBTEST.COM for oryx/127.23.65.125@KRBTEST.COM
I20260810 18:11:53.441840 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.434539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.0.0.1:59650)
0810 18:11:53.434894 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.434906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.434970 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.435177 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.435179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.435311 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.435419 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.435423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.436328 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.436331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.436876 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.436884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.437254 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.437256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.437337 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.437768 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.437780 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.438947 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.440482 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.440485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.440488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.440708 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.441014 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.441016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.441018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.441096 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.441436 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.441438 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.441523 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.441602 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.441707 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260810 18:11:53.441869 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.434649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.23.65.125:38417)
0810 18:11:53.434937 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.434941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.434954 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:53.435048 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.435050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.435091 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.435154 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.435509 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.436225 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.436966 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.437179 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.437249 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.439086 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.439089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.439091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.439115 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.440380 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.440833 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.440836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.440837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.440935 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.441162 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.441164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.441166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.441352 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.441425 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.441642 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.441740 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":111,"threads_started":1}
I20260810 18:11:53.442612 2583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.124@KRBTEST.COM for oryx/127.23.65.124@KRBTEST.COM
I20260810 18:11:53.450416 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.443119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.0.0.1:53590)
0810 18:11:53.443214 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.443227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.443298 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.443475 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.443477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.443651 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.443725 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.443730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.445287 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.445291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.445844 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.445854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.446257 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.446259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.446331 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.446684 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.446693 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.447838 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.449296 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.449298 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.449299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.449469 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.449680 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.449682 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.449683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.449745 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.450024 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.450026 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.450096 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.450166 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.450256 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1}
I20260810 18:11:53.450435 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.443187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.23.65.124:36711)
0810 18:11:53.443295 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.443298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.443310 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:11:53.443353 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.443355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.443399 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.443457 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.443847 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.445152 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.445946 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.446180 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.446252 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.447969 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.447972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.447974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.447994 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.449200 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.449536 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.449539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.449540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.449619 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.449809 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.449811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.449812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.449963 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.450031 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.450217 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.450304 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:11:53.453289 2583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Bootstrap starting.
I20260810 18:11:53.454897 1767 heartbeater.cc:503] Master 127.23.65.126:46139 was elected leader, sending a full tablet report...
I20260810 18:11:53.455809 2583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:53.456634 2583 log.cc:824] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:53.458500 2583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: No bootstrap required, opened a new log
I20260810 18:11:53.461547 2583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } }
I20260810 18:11:53.461756 2583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:53.461797 2583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ee363451724bd89da561e507770d1c, State: Initialized, Role: FOLLOWER
I20260810 18:11:53.462168 2583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } }
I20260810 18:11:53.462616 2592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } } }
I20260810 18:11:53.462738 2592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:53.463289 2583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:11:53.468062 2608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:53.468128 2608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:53.471109 2358 master_runner.cc:428] Detected that this master 32ee363451724bd89da561e507770d1c is joining an existing cluster
I20260810 18:11:53.471170 2358 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.65.124:36711
I20260810 18:11:53.481945 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 1565
I20260810 18:11:53.482568 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.473992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.23.65.126:46139)
0810 18:11:53.474185 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.474189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.474408 (+ 219us) server_negotiation.cc:407] Connection header received
0810 18:11:53.474519 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.474522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.474573 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.474653 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.475244 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.476123 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.477366 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.477702 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.477779 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.479204 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.479207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.479209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.479232 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.480643 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.481123 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.481127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.481129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.481226 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.481500 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.481504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.481505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.481758 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.481850 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.482306 (+ 456us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.482426 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 18:11:53.483022 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.473528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:44510)
0810 18:11:53.474330 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.474342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.474436 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.474691 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.474694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.474944 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.475093 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.475101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.476270 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.476273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.477213 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.477224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.477956 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.477960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.478052 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.478594 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.478609 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.479066 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.480737 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.480740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.480743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.481014 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.481326 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.481328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.481330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.481407 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.481926 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.481941 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.482107 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.482754 (+ 647us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.482877 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260810 18:11:53.522727 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.473670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.23.65.125:38417)
0810 18:11:53.473799 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.473803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.474694 (+ 891us) server_negotiation.cc:407] Connection header received
0810 18:11:53.474836 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.474839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.474891 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.474960 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.475576 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.476378 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.478006 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.478298 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.478373 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.519113 (+ 40740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.519119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.519122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.519160 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.520794 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.521305 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.521309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.521311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.521413 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.521705 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.521708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.521710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.521934 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.522014 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.522392 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.522551 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:11:53.523321 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.473572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.0.0.1:59658)
0810 18:11:53.474623 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.474636 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.474754 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.475074 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.475077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.475308 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.475419 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.475426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.476509 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.476512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.477888 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.477896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.478527 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.478530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.478613 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.518047 (+ 39434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.518069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.518871 (+ 802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.520937 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.520941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.520943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.521189 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.521522 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.521525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.521527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.521611 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.522092 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.522095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.522196 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.522958 (+ 762us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.523121 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":28,"thread_start_us":81,"threads_started":1}
I20260810 18:11:53.527307 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.474048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.23.65.124:36711)
0810 18:11:53.474149 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.474153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.475030 (+ 877us) server_negotiation.cc:407] Connection header received
0810 18:11:53.475151 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.475154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.475188 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.475229 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.475755 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.476454 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.477461 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.477796 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.477862 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.523715 (+ 45853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.523719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.523721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.523743 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.525680 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.526108 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.526111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.526113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.526265 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.526527 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.526530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.526532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.526711 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.526775 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.527051 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.527160 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":30}
I20260810 18:11:53.527697 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:34559
--local_ip_for_outbound_sockets=127.23.65.65
--tserver_master_addrs=127.23.65.126:46139
--webserver_port=45415
--webserver_interface=127.23.65.65
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab
--principal=oryx/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.65.126:46139,127.23.65.125:38417
--tserver_master_addrs=127.23.65.126:46139,127.23.65.125:38417,127.23.65.124:36711 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:11:53.527817 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.474031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.0.0.1:53602)
0810 18:11:53.474967 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.474978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.475073 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.475352 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.475355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.475521 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.475656 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.475663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.476555 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.476557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.477347 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.477358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.477999 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.478002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.478084 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.518250 (+ 40166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.518270 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.523584 (+ 5314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.525791 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.525794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.525796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.526016 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.526371 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.526373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.526376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.526451 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.526829 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.526832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.526909 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.527437 (+ 528us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.527586 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":92,"threads_started":1}
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385512, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.126@KRBTEST.COM for oryx/127.23.65.124@KRBTEST.COM
I20260810 18:11:53.536587 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.525224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.0.0.1:53616)
0810 18:11:53.525455 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.525470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.525545 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.526302 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.526305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.526504 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.526629 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.526637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.528814 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.528818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.529559 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.529570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.530326 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.530330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.530409 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.530827 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.530840 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.532354 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.534687 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.534691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.534693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.534949 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.535404 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.535407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.535409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.535503 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.536048 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.536051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.536151 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.536253 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.536406 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260810 18:11:53.537444 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.525278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.23.65.124:36711)
0810 18:11:53.525904 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.525938 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.525954 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:11:53.526097 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.526101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.526155 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.526752 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.526995 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.528675 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.529730 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.530190 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.530603 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.532518 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.532522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.532524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.532551 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.534554 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.535047 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.535050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.535052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.535304 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.535628 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.535632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.535634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.535953 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.536060 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.536330 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.536447 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":99,"threads_started":1}
I20260810 18:11:53.538399 1466 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.65.124:36711: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "32ee363451724bd89da561e507770d1c"
member_type: NON_VOTER
last_known_addr {
host: "127.23.65.124"
port: 36711
}
attrs {
promote: true
}
}
dest_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e"
cas_config_opid_index: 5
I20260810 18:11:53.539108 1466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } }
I20260810 18:11:53.540530 2338 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 32ee363451724bd89da561e507770d1c (127.23.65.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.542033 2338 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.65.124:36711
I20260810 18:11:53.542237 2338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.542330 2338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:53.542730 2358 master.cc:537] Master@127.23.65.124:36711 shutting down...
W20260810 18:11:53.544987 1448 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e -> Peer 32ee363451724bd89da561e507770d1c (127.23.65.124:36711): Couldn't send request to peer 32ee363451724bd89da561e507770d1c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:11:53.546576 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.540668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.0.0.1:59668)
0810 18:11:53.540979 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.540994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.541068 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.541470 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.541474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.541491 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:53.541570 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.541575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.543112 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.543118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.544439 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.544452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.545751 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.545755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.546252 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.546303 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.546389 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.546456 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260810 18:11:53.547005 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.540916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.23.65.125:38417)
0810 18:11:53.541043 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.541047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.541062 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:53.541188 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.541191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.541253 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.541350 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:53.541682 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.542477 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.544585 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.545244 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.545335 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.545488 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.546811 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.546891 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:11:53.548246 2420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 LEARNER]: Refusing update from remote peer 2b729eb5df9e496cbbfeaa1fc28ed39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 18:11:53.548326 2358 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:53.548604 2358 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:53.548694 2358 tablet_replica.cc:330] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: stopping tablet replica
I20260810 18:11:53.548671 2337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:11:53.550547 2420 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 32ee363451724bd89da561e507770d1c (127.23.65.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.552976 2467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b729eb5df9e496cbbfeaa1fc28ed39e. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.553102 2467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: This master's current role is: LEARNER
I20260810 18:11:53.553193 2467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: NON_VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: true } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.553267 2467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: This master's current role is: LEARNER
I20260810 18:11:53.554510 2337 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: attempting to promote NON_VOTER 510ce43378724dd8816f3b2dd123516f to VOTER
I20260810 18:11:53.554956 2337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } }
I20260810 18:11:53.556139 2417 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 LEARNER]: Refusing update from remote peer 2b729eb5df9e496cbbfeaa1fc28ed39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 18:11:53.556459 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.556172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.0.0.1:53626)
0810 18:11:53.556313 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.556379 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 18:11:53.557440 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.557201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.0.0.1:53642)
0810 18:11:53.557333 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.557379 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 18:11:53.557564 1448 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e -> Peer 32ee363451724bd89da561e507770d1c (127.23.65.124:36711): Couldn't send request to peer 32ee363451724bd89da561e507770d1c. Status: Network error: Client connection negotiation failed: client connection to 127.23.65.124:36711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:11:53.557691 2337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:11:53.559381 2335 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 510ce43378724dd8816f3b2dd123516f (127.23.65.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.560899 2335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.560999 2335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:53.561120 2335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.561180 2335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:53.561507 2417 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 510ce43378724dd8816f3b2dd123516f (127.23.65.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.563171 2625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.563303 2625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:53.566301 2358 master.cc:552] Master@127.23.65.124:36711 shutdown complete.
I20260810 18:11:53.566354 2358 master_runner.cc:305] Clearing existing system tablet
I20260810 18:11:53.566586 2358 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:53.566627 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:53.567937 2358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 18:11:53.568799 2629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.569031 2358 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:53.569221 2358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 18:11:53.569288 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal
uuid: "32ee363451724bd89da561e507770d1c"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "1774a4f92adc949876a44c370897bd53"
server_key_iv: "5951a52d8e235041e3f31e18e89f036d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.571308 2358 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:53.573724 2358 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 18:11:53.573853 2358 log.cc:1196] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 18:11:53.574127 2358 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Deleting consensus metadata
I20260810 18:11:53.574319 2358 master_runner.cc:315] Copying system tablet from 127.23.65.126:46139
I20260810 18:11:53.575652 2358 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: tablet copy: Beginning tablet copy session from remote peer at address 127.23.65.126:46139
I20260810 18:11:53.587932 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.576210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:44514)
0810 18:11:53.579810 (+ 3600us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.579826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.579952 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.580238 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.580241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.580468 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.580574 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.580581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.581614 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.581618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.582495 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.582507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.583080 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.583085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.583174 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.583846 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.583861 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.584284 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.586202 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.586206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.586208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.586471 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.586767 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.586770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.586772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.586866 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.587294 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.587298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.587407 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.587580 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.587745 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3523,"thread_start_us":96,"threads_started":1}
I20260810 18:11:53.588274 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.576295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.23.65.126:46139)
0810 18:11:53.576413 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.576417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.579921 (+ 3504us) server_negotiation.cc:407] Connection header received
0810 18:11:53.580062 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.580066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.580122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.580207 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.580695 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.581490 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.582629 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.582955 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.583063 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.584455 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.584459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.584462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.584489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.586070 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.586568 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.586572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.586574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.586677 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.586957 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.586961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.586963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.587190 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.587295 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.587698 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.587829 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:11:53.588792 1495 tablet_copy_service.cc:140] P 2b729eb5df9e496cbbfeaa1fc28ed39e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 32ee363451724bd89da561e507770d1c ({username='oryx', principal='oryx/127.23.65.124@KRBTEST.COM'} at 127.0.0.1:44514)
I20260810 18:11:53.588886 1495 tablet_copy_service.cc:161] P 2b729eb5df9e496cbbfeaa1fc28ed39e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 32ee363451724bd89da561e507770d1c at {username='oryx', principal='oryx/127.23.65.124@KRBTEST.COM'} at 127.0.0.1:44514: session id = 32ee363451724bd89da561e507770d1c-00000000000000000000000000000000
I20260810 18:11:53.589949 1495 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:11:53.591180 2358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 18:11:53.593873 2358 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: tablet copy: Starting download of 0 data blocks...
I20260810 18:11:53.594172 2358 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: tablet copy: Starting download of 1 WAL segments...
I20260810 18:11:53.595896 2358 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:11:53.597752 1495 tablet_copy_service.cc:342] P 2b729eb5df9e496cbbfeaa1fc28ed39e: Request end of tablet copy session 32ee363451724bd89da561e507770d1c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.65.124@KRBTEST.COM'} at 127.0.0.1:44514
I20260810 18:11:53.597833 1495 tablet_copy_service.cc:433] P 2b729eb5df9e496cbbfeaa1fc28ed39e: ending tablet copy session 32ee363451724bd89da561e507770d1c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 32ee363451724bd89da561e507770d1c
I20260810 18:11:53.605600 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:53.607313 2637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.607658 2638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:53.607904 2358 server_base.cc:1034] running on GCE node
W20260810 18:11:53.607919 2641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:53.608213 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:53.608556 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:53.609746 2358 hybrid_clock.cc:648] HybridClock initialized: now 1786385513609730 us; error 33 us; skew 500 ppm
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:53.611464 2358 init.cc:377] Logged in from keytab as oryx/127.23.65.124@KRBTEST.COM (short username oryx)
I20260810 18:11:53.611959 2358 webserver.cc:460] Webserver started at http://127.23.65.124:46161/ using document root <none> and password file <none>
I20260810 18:11:53.612147 2358 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:53.612200 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:53.613552 2358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:53.614543 2651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.614840 2358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:53.614923 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal
uuid: "32ee363451724bd89da561e507770d1c"
format_stamp: "Formatted at 2026-08-10 18:11:53 on dist-test-slave-jpj1"
server_key: "1774a4f92adc949876a44c370897bd53"
server_key_iv: "5951a52d8e235041e3f31e18e89f036d"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.615013 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:53.625720 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:53.627692 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:53.633610 2358 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.124:36711
I20260810 18:11:53.634251 2727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.124:36711 every 8 connection(s)
I20260810 18:11:53.634416 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
I20260810 18:11:53.635890 2728 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:11:53.636711 2728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Bootstrap starting.
I20260810 18:11:53.645635 2728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:11:53.645895 2728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Bootstrap complete.
I20260810 18:11:53.646363 2728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } }
I20260810 18:11:53.646481 2728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 32ee363451724bd89da561e507770d1c, State: Initialized, Role: LEARNER
I20260810 18:11:53.646584 2728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } }
I20260810 18:11:53.647023 2728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:53.647125 2733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.647320 2733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: This master's current role is: LEARNER
I20260810 18:11:53.651993 2747 catalog_manager.cc:1259] Loaded cluster ID: 832069d7906941fa973f3b2ee8c3eb56
I20260810 18:11:53.652047 2747 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: loading cluster ID for follower catalog manager: success
I20260810 18:11:53.653497 2747 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: acquiring CA information for follower catalog manager: success
I20260810 18:11:53.654564 2747 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:11:53.669695 1974 heartbeater.cc:503] Master 127.23.65.126:46139 was elected leader, sending a full tablet report...
W20260810 18:11:53.683373 2622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:53.683703 2622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:53.683763 2622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:53.687939 2622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:53.687989 2622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:53.688056 2622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:53.693554 2622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab
--principal=oryx/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:34559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=45415
--tserver_master_addrs=127.23.65.126:46139,127.23.65.125:38417,127.23.65.124:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:53.694643 2622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:53.695693 2622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:53.706516 2759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.706475 2756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.706475 2754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:53.706722 2622 server_base.cc:1034] running on GCE node
I20260810 18:11:53.707252 2622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:53.707947 2622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:53.709100 2622 hybrid_clock.cc:648] HybridClock initialized: now 1786385513709072 us; error 42 us; skew 500 ppm
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:53.712342 2622 init.cc:377] Logged in from keytab as oryx/127.23.65.65@KRBTEST.COM (short username oryx)
I20260810 18:11:53.713291 2622 webserver.cc:460] Webserver started at http://127.23.65.65:45415/ using document root <none> and password file <none>
I20260810 18:11:53.713855 2622 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:53.713938 2622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:53.718679 2622 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260810 18:11:53.721131 2772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.722450 2622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:11:53.722558 2622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "e92666cfe2574cff8a51c370ad089327"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "dcafd4c3e777377e69ef20aff1f574b0"
server_key_iv: "e74bf55a0ef009f660be56f551252ff0"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.723250 2622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:53.733220 2622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:53.736297 2622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:53.736495 2622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:53.737294 2622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:53.738305 2622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:53.738364 2622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.738408 2622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:53.738437 2622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.757043 2622 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:34559
I20260810 18:11:53.757097 2956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:34559 every 8 connection(s)
I20260810 18:11:53.758119 2622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:53.766105 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 2622
I20260810 18:11:53.766302 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 1777
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.124@KRBTEST.COM
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.125@KRBTEST.COM
I20260810 18:11:53.776925 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.760128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:51739 (local address 127.23.65.124:36711)
0810 18:11:53.760405 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.760410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.761852 (+ 1442us) server_negotiation.cc:407] Connection header received
0810 18:11:53.762518 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.762522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.762580 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.762662 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.764123 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.765228 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.766670 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.766992 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.767063 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.772081 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.772084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.772087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.772114 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.774072 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.774852 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.774856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.774857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.774956 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.775469 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.775473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.775474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.775734 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.775807 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.776618 (+ 811us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.776756 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260810 18:11:53.776924 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.760843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:39075 (local address 127.23.65.125:38417)
0810 18:11:53.760967 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.760971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.761972 (+ 1001us) server_negotiation.cc:407] Connection header received
0810 18:11:53.762552 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.762555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.762611 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.762670 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.765114 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.765720 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.766808 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.767231 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.767334 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.772485 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.772489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.772491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.772521 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.774186 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.774852 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.774856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.774857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.775054 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.775469 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.775473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.775474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.775733 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.775807 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.776618 (+ 811us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.776748 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:11:53.777937 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.760468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:33795 (local address 127.23.65.126:46139)
0810 18:11:53.760614 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.760619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.761480 (+ 861us) server_negotiation.cc:407] Connection header received
0810 18:11:53.762551 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.762555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.762609 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.762662 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.764162 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.764980 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.766967 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.767284 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.767356 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.771939 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:53.771944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:53.771946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:53.771976 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:53.774375 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.775011 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.775015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.775017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.775128 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:53.775530 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:53.775568 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:53.775570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:53.775842 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:53.775925 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.777603 (+ 1678us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.777760 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:11:53.778627 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.760577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.65:33795)
0810 18:11:53.761296 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.761359 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.762305 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.762860 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.762868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.763325 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.763996 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.764006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.765111 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.765114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.766848 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.766857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.767504 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.767507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.767698 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.769298 (+ 1600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.769309 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.771748 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.774526 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.774530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.774533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.774875 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.775264 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.775267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.775280 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.775395 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.776008 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.776011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.776320 (+ 309us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.778106 (+ 1786us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.778290 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":464,"spinlock_wait_cycles":81408,"thread_start_us":297,"threads_started":1}
I20260810 18:11:53.778620 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.760577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.23.65.65:51739)
0810 18:11:53.761757 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.761771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.762313 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.762859 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.762867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.763318 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.763986 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.764005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.765330 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.765334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.766541 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.766552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.767256 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.767260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.767450 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.769237 (+ 1787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.769262 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.771920 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.774212 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.774218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.774237 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.774733 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.775170 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.775173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.775174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.775355 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.775894 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.775901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.776308 (+ 407us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.777092 (+ 784us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.777514 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":214,"spinlock_wait_cycles":116352,"thread_start_us":888,"threads_started":1}
I20260810 18:11:53.778620 2969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.760727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.23.65.65:39075)
0810 18:11:53.761879 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.761892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.762318 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.762860 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.762868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.763322 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.763993 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.764006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.765856 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.765859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.766664 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.766676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.767517 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.767522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.767707 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.769421 (+ 1714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.769438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.772326 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:53.774290 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.774293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.774295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.774733 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.775136 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:53.775139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:53.775141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:53.775355 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:53.775894 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:53.775901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:53.776307 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.777308 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.777577 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"mutex_wait_us":231,"spinlock_wait_cycles":1763584,"thread_start_us":933,"threads_started":1}
I20260810 18:11:53.780040 2962 heartbeater.cc:347] Connected to a master server at 127.23.65.125:38417
I20260810 18:11:53.780295 2962 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:53.780794 2962 heartbeater.cc:511] Master 127.23.65.125:38417 requested a full tablet report, sending...
I20260810 18:11:53.782423 2397 ts_manager.cc:194] Registered new tserver with Master: e92666cfe2574cff8a51c370ad089327 (127.23.65.65:34559)
I20260810 18:11:53.785983 2960 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46139
I20260810 18:11:53.786082 2960 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:53.786167 2959 heartbeater.cc:347] Connected to a master server at 127.23.65.124:36711
I20260810 18:11:53.786247 2959 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:53.786257 2960 heartbeater.cc:511] Master 127.23.65.126:46139 requested a full tablet report, sending...
I20260810 18:11:53.786392 2959 heartbeater.cc:511] Master 127.23.65.124:36711 requested a full tablet report, sending...
I20260810 18:11:53.787106 1466 ts_manager.cc:194] Re-registered known tserver with Master: e92666cfe2574cff8a51c370ad089327 (127.23.65.65:34559)
I20260810 18:11:53.787612 2677 ts_manager.cc:194] Registered new tserver with Master: e92666cfe2574cff8a51c370ad089327 (127.23.65.65:34559)
I20260810 18:11:53.787946 1466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:33795
I20260810 18:11:53.792531 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:40939
--local_ip_for_outbound_sockets=127.23.65.66
--tserver_master_addrs=127.23.65.126:46139
--webserver_port=46623
--webserver_interface=127.23.65.66
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab
--principal=oryx/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.65.126:46139,127.23.65.125:38417
--tserver_master_addrs=127.23.65.126:46139,127.23.65.125:38417,127.23.65.124:36711 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:11:53.887207 2177 heartbeater.cc:503] Master 127.23.65.126:46139 was elected leader, sending a full tablet report...
W20260810 18:11:53.908926 2970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:53.909194 2970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:53.909255 2970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:53.913307 2970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:53.913384 2970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:53.913475 2970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:53.919137 2970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab
--principal=oryx/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:40939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=46623
--tserver_master_addrs=127.23.65.126:46139,127.23.65.125:38417,127.23.65.124:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:53.920333 2970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:53.921340 2970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:53.930917 2979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.930992 2982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:53.930970 2978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:53.931742 2970 server_base.cc:1034] running on GCE node
I20260810 18:11:53.932122 2970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:53.932739 2970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:53.933916 2970 hybrid_clock.cc:648] HybridClock initialized: now 1786385513933891 us; error 35 us; skew 500 ppm
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:53.936801 2970 init.cc:377] Logged in from keytab as oryx/127.23.65.66@KRBTEST.COM (short username oryx)
I20260810 18:11:53.937619 2970 webserver.cc:460] Webserver started at http://127.23.65.66:46623/ using document root <none> and password file <none>
I20260810 18:11:53.938148 2970 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:53.938233 2970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:53.942044 2970 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:53.944207 2996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.945431 2970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:53.945534 2970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "70d63c94c94d4c77995cbf795453d93e"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "1bdf49d73afbecc95effea584c565f76"
server_key_iv: "0574d82074bcc60f0aaff8d64bf76b9c"
server_key_version: "encryptionkey@0"
I20260810 18:11:53.946084 2970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:53.956123 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.952429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.23.65.124:36711)
0810 18:11:53.952554 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.952558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.952574 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:11:53.952740 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.952743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.952811 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.952901 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:53.953128 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.953655 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.955087 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.955613 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.955699 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.955848 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:53.955945 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 18:11:53.956016 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:11:53.956143 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.952323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.0.0.1:53644)
0810 18:11:53.952451 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.952467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.952572 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.952917 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.952920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.952934 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:53.953038 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.953043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.953739 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.953742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.954937 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.954945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.955720 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.955723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.955821 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.955848 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:11:53.955911 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 18:11:53.955993 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 18:11:53.957093 2692 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 18:11:53.957549 2733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b729eb5df9e496cbbfeaa1fc28ed39e. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:53.957681 2733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: This master's current role is: LEARNER
I20260810 18:11:53.970679 2970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:53.973510 2970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:53.973663 2970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:53.974242 2970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:53.975111 2970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:53.975160 2970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.975203 2970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:53.975224 2970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:53.983578 2970 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:40939
I20260810 18:11:53.983639 3165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:40939 every 8 connection(s)
I20260810 18:11:53.984561 2970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:53.989756 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 2970
I20260810 18:11:53.989940 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 1982
Aug 10 18:11:53 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:54.004033 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.986273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:40885 (local address 127.23.65.126:46139)
0810 18:11:53.986507 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.986511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.987799 (+ 1288us) server_negotiation.cc:407] Connection header received
0810 18:11:53.988703 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.988706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.988767 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.988838 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.994482 (+ 5644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.995254 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.996475 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.996900 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.997029 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.000162 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.000166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.000169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.000201 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.001586 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.002171 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.002175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.002177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.002261 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.002625 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.002629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.002631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.002875 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.002960 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.003692 (+ 732us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.003850 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
Aug 10 18:11:54 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.125@KRBTEST.COM
Aug 10 18:11:54 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385513, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.124@KRBTEST.COM
I20260810 18:11:54.006156 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.986730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.66:40885)
0810 18:11:53.987710 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.987722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.988507 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.989018 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.989025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.989491 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.994303 (+ 4812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.994315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.995383 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.995388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.996344 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.996355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.997017 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.997019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.997096 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:53.997664 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:53.997691 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:53.999951 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:54.001718 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.001724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.001742 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.002068 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.002347 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.002349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.002350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.002517 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.003029 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:54.003036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:54.003380 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.004410 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.004883 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":91,"spinlock_wait_cycles":33792,"thread_start_us":272,"threads_started":1}
I20260810 18:11:54.007849 3168 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46139
I20260810 18:11:54.008117 3168 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:54.008742 3168 heartbeater.cc:511] Master 127.23.65.126:46139 requested a full tablet report, sending...
I20260810 18:11:54.009331 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.986263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:47733 (local address 127.23.65.125:38417)
0810 18:11:53.986398 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.986402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.987569 (+ 1167us) server_negotiation.cc:407] Connection header received
0810 18:11:53.988720 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.988724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.988775 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.988828 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.990373 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.991028 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.992470 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.992738 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.992797 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.005387 (+ 12590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.005392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.005395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.005427 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.007284 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.007925 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.007930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.007931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.008079 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.008424 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.008428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.008429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.008652 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.008728 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.009065 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.009187 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:11:54.009848 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.986695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:57853 (local address 127.23.65.124:36711)
0810 18:11:53.986824 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:53.986828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:53.988028 (+ 1200us) server_negotiation.cc:407] Connection header received
0810 18:11:53.988720 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:53.988724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:53.988775 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:53.988848 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:53.994413 (+ 5565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.995206 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.003438 (+ 8232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.003806 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.003894 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.006328 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.006332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.006334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.006363 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.008009 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.008496 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.008500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.008501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.008596 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.008902 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.008905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.008907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.009117 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.009198 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.009549 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.009705 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:11:54.010036 1466 ts_manager.cc:194] Re-registered known tserver with Master: 70d63c94c94d4c77995cbf795453d93e (127.23.65.66:40939)
I20260810 18:11:54.010740 1466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:40885
I20260810 18:11:54.010879 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.986730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.23.65.66:47733)
0810 18:11:53.987400 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.987450 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.988514 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.989019 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.989027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.989488 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.990214 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.990230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.991181 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.991184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.992297 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.992307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.993029 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.993033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:53.993213 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.003215 (+ 10002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:54.003230 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:54.005189 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:54.007430 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.007433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.007435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.007802 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.008191 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.008194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.008196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.008321 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.008801 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:54.008804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:54.008890 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.009759 (+ 869us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.010351 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":92,"thread_start_us":296,"threads_started":1}
I20260810 18:11:54.010970 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:53.986730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.23.65.66:57853)
0810 18:11:53.987926 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:53.987938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:53.988512 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:53.989019 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:53.989027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:53.989505 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:53.994276 (+ 4771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:53.994287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:53.995368 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:53.995371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.003258 (+ 7887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.003267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.004058 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.004062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.004162 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.004631 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:54.004647 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:54.006137 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:54.008132 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.008135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.008137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.008380 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.008703 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.008706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.008708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.008793 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.009255 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:54.009258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:54.009360 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.009991 (+ 631us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.010450 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":11705,"spinlock_wait_cycles":22784,"thread_start_us":706,"threads_started":1}
I20260810 18:11:54.011003 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:40235
--local_ip_for_outbound_sockets=127.23.65.67
--tserver_master_addrs=127.23.65.126:46139
--webserver_port=37785
--webserver_interface=127.23.65.67
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab
--principal=oryx/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.65.126:46139,127.23.65.125:38417
--tserver_master_addrs=127.23.65.126:46139,127.23.65.125:38417,127.23.65.124:36711 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:11:54.011608 3169 heartbeater.cc:347] Connected to a master server at 127.23.65.125:38417
I20260810 18:11:54.011699 3169 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:54.011878 3169 heartbeater.cc:511] Master 127.23.65.125:38417 requested a full tablet report, sending...
I20260810 18:11:54.012002 3166 heartbeater.cc:347] Connected to a master server at 127.23.65.124:36711
I20260810 18:11:54.012087 3166 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:54.012275 3166 heartbeater.cc:511] Master 127.23.65.124:36711 requested a full tablet report, sending...
I20260810 18:11:54.012633 2397 ts_manager.cc:194] Registered new tserver with Master: 70d63c94c94d4c77995cbf795453d93e (127.23.65.66:40939)
I20260810 18:11:54.013401 2677 ts_manager.cc:194] Registered new tserver with Master: 70d63c94c94d4c77995cbf795453d93e (127.23.65.66:40939)
W20260810 18:11:54.125903 3175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:54.126165 3175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:54.126209 3175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:54.130290 3175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:54.130344 3175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:54.130399 3175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:54.135460 3175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab
--principal=oryx/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:40235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=37785
--tserver_master_addrs=127.23.65.126:46139,127.23.65.125:38417,127.23.65.124:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:54.136456 3175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:54.137444 3175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:54.147068 3187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:54.147078 3181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:54.147059 3183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:54.147343 3175 server_base.cc:1034] running on GCE node
I20260810 18:11:54.147887 3175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:54.148556 3175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:54.149737 3175 hybrid_clock.cc:648] HybridClock initialized: now 1786385514149713 us; error 36 us; skew 500 ppm
Aug 10 18:11:54 dist-test-slave-jpj1 krb5kdc[1407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385514, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:54.152694 3175 init.cc:377] Logged in from keytab as oryx/127.23.65.67@KRBTEST.COM (short username oryx)
I20260810 18:11:54.153626 3175 webserver.cc:460] Webserver started at http://127.23.65.67:37785/ using document root <none> and password file <none>
I20260810 18:11:54.154204 3175 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:54.154289 3175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:54.158442 3175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260810 18:11:54.160537 3196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:54.161789 3175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:54.161900 3175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "56e697639a4d4f519a9fa399614b49b8"
format_stamp: "Formatted at 2026-08-10 18:11:52 on dist-test-slave-jpj1"
server_key: "c760323f66ba2018aa9b62b416574023"
server_key_iv: "fa0acb02be65d198b1472dbbf4d6eeab"
server_key_version: "encryptionkey@0"
I20260810 18:11:54.162477 3175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:54.178124 3175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:54.181219 3175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:54.181411 3175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:54.181998 3175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:11:54.182847 3175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:54.182894 3175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:54.182938 3175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:54.182967 3175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:54.193404 3175 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:40235
I20260810 18:11:54.193429 3368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:40235 every 8 connection(s)
I20260810 18:11:54.194715 3175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:54.198632 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 3175
Aug 10 18:11:54 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385514, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.124@KRBTEST.COM
Aug 10 18:11:54 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385514, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.125@KRBTEST.COM
Aug 10 18:11:54 dist-test-slave-jpj1 krb5kdc[1407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385514, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:54.213340 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.197491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:37085 (local address 127.23.65.124:36711)
0810 18:11:54.197614 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.197618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.198592 (+ 974us) server_negotiation.cc:407] Connection header received
0810 18:11:54.199325 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.199329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.199386 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.199438 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.201437 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.202274 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.204837 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.205161 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.205242 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.207807 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.207811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.207814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.207840 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.209393 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.210012 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.210016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.210018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.210111 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.210524 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.210528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.210530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.210745 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.210841 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.213057 (+ 2216us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.213177 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:11:54.213415 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.197544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:32955 (local address 127.23.65.125:38417)
0810 18:11:54.197659 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.197664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.198123 (+ 459us) server_negotiation.cc:407] Connection header received
0810 18:11:54.199327 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.199330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.199386 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.199438 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.201562 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.202329 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.204203 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.204543 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.204632 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.208534 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.208538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.208540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.208562 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.210939 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.211409 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.211413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.211415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.211502 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.212503 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.212506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.212508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.212725 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.212804 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.213149 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.213262 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:11:54.213575 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.197284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.23.65.67:37085)
0810 18:11:54.198481 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:54.198494 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:54.199078 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:54.199863 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:54.199872 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:54.200447 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:54.201197 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.201220 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.202452 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.202455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.204702 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.204713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.205373 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.205377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.205471 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.205898 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:54.205913 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:54.207620 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:54.209497 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.209503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.209518 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.209892 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.210213 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.210216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.210218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.210408 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.210899 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:54.210905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:54.211226 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.211449 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.211824 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"mutex_wait_us":1261,"spinlock_wait_cycles":29952,"thread_start_us":230,"threads_started":1}
I20260810 18:11:54.213923 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.197283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.23.65.67:32955)
0810 18:11:54.197970 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:54.198024 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:54.199074 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:54.199863 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:54.199872 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:54.200690 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:54.201421 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.201432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.202453 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.202455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.204071 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.204081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.204827 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.204830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.205044 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.205661 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:54.205684 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:54.208000 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:54.211036 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.211039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.211041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.211291 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.211651 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.211654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.211656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.212387 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.212865 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:54.212867 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:54.212959 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.213608 (+ 649us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.213748 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":132,"thread_start_us":749,"threads_started":1}
I20260810 18:11:54.214924 3371 heartbeater.cc:347] Connected to a master server at 127.23.65.125:38417
I20260810 18:11:54.215104 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.201248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.23.65.125:38417)
0810 18:11:54.202907 (+ 1659us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.202911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.203487 (+ 576us) server_negotiation.cc:407] Connection header received
0810 18:11:54.203524 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.203527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.203599 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.203669 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.205890 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.206675 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.208037 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.208373 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.208470 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.210091 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.210095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.210097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.210127 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.212096 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.212674 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.212677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.212679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.212770 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.213081 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.213085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.213086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.213278 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.213956 (+ 678us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.214754 (+ 798us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.214876 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1588,"spinlock_wait_cycles":3328,"thread_start_us":89,"threads_started":1}
I20260810 18:11:54.215193 3371 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:54.215566 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.200695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.23.65.126:46139)
0810 18:11:54.200955 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.200959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.201743 (+ 784us) server_negotiation.cc:407] Connection header received
0810 18:11:54.202071 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.202090 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.202143 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.202202 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.203247 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.204043 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.205160 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.205989 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.206086 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.211870 (+ 5784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.211874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.211876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.211903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.213577 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.214098 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.214102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.214103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.214195 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.214508 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.214512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.214514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.214725 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.214836 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.215250 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.215376 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260810 18:11:54.215811 3371 heartbeater.cc:511] Master 127.23.65.125:38417 requested a full tablet report, sending...
I20260810 18:11:54.214932 3369 heartbeater.cc:347] Connected to a master server at 127.23.65.124:36711
I20260810 18:11:54.216459 3369 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:54.216631 3369 heartbeater.cc:511] Master 127.23.65.124:36711 requested a full tablet report, sending...
I20260810 18:11:54.217406 2397 ts_manager.cc:194] Registered new tserver with Master: 56e697639a4d4f519a9fa399614b49b8 (127.23.65.67:40235)
I20260810 18:11:54.217468 2677 ts_manager.cc:194] Registered new tserver with Master: 56e697639a4d4f519a9fa399614b49b8 (127.23.65.67:40235)
I20260810 18:11:54.220717 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.201253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.23.65.124:36711)
0810 18:11:54.203048 (+ 1795us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.203052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.203435 (+ 383us) server_negotiation.cc:407] Connection header received
0810 18:11:54.203478 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.203481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.203528 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.203635 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.204512 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.205228 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.208420 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.208751 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.208851 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.216070 (+ 7219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.216074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.216076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.216102 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.218709 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.219192 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.219195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.219196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.219309 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.219658 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.219661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.219663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.219884 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.220007 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.220329 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.220558 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1723,"thread_start_us":79,"threads_started":1}
I20260810 18:11:54.219376 1466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:11:54.221817 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.196886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:41495 (local address 127.23.65.126:46139)
0810 18:11:54.197028 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.197033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.198813 (+ 1780us) server_negotiation.cc:407] Connection header received
0810 18:11:54.199349 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.199353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.199416 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.199474 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.201639 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.202459 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.207061 (+ 4602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.207378 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.207477 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.218347 (+ 10870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.218351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.218354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.218375 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.219994 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.220664 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.220667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.220668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.220760 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.221058 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.221062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.221064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.221241 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.221323 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.221582 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.221687 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:11:54.221829 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.197284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.67:41495)
0810 18:11:54.198717 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:54.198728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:54.199084 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:54.199865 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:54.199873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:54.200454 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:54.201497 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.201505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.202611 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.202614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.206951 (+ 4337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.206961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.208231 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.208236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.209560 (+ 1324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.210569 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:54.210582 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:54.218234 (+ 7652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:54.220100 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.220103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.220105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.220547 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.220872 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:54.220875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:54.220877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:54.220959 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:54.221334 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:54.221337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:54.221423 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.221499 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.221653 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"mutex_wait_us":1629,"thread_start_us":121,"threads_started":1}
I20260810 18:11:54.222657 3370 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46139
I20260810 18:11:54.222750 3370 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:54.222910 3370 heartbeater.cc:511] Master 127.23.65.126:46139 requested a full tablet report, sending...
I20260810 18:11:54.223493 1464 ts_manager.cc:194] Re-registered known tserver with Master: 56e697639a4d4f519a9fa399614b49b8 (127.23.65.67:40235)
W20260810 18:11:54.223609 1466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:54.224140 1464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:41495
I20260810 18:11:54.226022 2338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: NON_VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: true } } }
I20260810 18:11:54.226143 2338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:54.229344 3399 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: attempting to promote NON_VOTER 32ee363451724bd89da561e507770d1c to VOTER
I20260810 18:11:54.229717 3399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } }
I20260810 18:11:54.230796 2692 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 LEARNER]: Refusing update from remote peer 2b729eb5df9e496cbbfeaa1fc28ed39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 18:11:54.231287 3398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 18:11:54.231978 2417 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Refusing update from remote peer 2b729eb5df9e496cbbfeaa1fc28ed39e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 18:11:54.232328 3398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 18:11:54.233434 3398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 32ee363451724bd89da561e507770d1c (127.23.65.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:54.234683 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:54.234773 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:54.234879 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:54.234942 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:54.235760 2338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:54.235862 2338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:54.235888 2692 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 32ee363451724bd89da561e507770d1c (127.23.65.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:54.236163 2417 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 32ee363451724bd89da561e507770d1c (127.23.65.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:54.237969 3412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:54.238092 2733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:54.238199 2733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:54.238250 3412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:54.248253 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.241388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:34559 (local address 127.0.0.1:55208)
0810 18:11:54.242459 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:54.242473 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:54.242630 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:54.243513 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:54.243516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:54.243558 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:54.243664 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.243670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.245167 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.245171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.246429 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.246437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.247822 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.247826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.247917 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.247943 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.248019 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.248105 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":105,"threads_started":1}
I20260810 18:11:54.248719 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.241829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.23.65.65:34559)
0810 18:11:54.242224 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.242244 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.242555 (+ 311us) server_negotiation.cc:407] Connection header received
0810 18:11:54.242788 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.242794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.243052 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.243330 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:54.243791 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.245027 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.246550 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.247422 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.247528 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.247711 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.248442 (+ 731us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.248551 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260810 18:11:54.249771 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.242646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.23.65.66:40939)
0810 18:11:54.243027 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.243041 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.243416 (+ 375us) server_negotiation.cc:407] Connection header received
0810 18:11:54.243503 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.243508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.243748 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.243894 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:54.244324 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.245412 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.246928 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.248027 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.248111 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.248246 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.249415 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.249541 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260810 18:11:54.250149 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.242222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:40939 (local address 127.0.0.1:52748)
0810 18:11:54.243236 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:54.243249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:54.243332 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:54.244088 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:54.244091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:54.244109 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:54.244194 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.244200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.245532 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.245535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.246794 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.246806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.248471 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.248474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.248571 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.248603 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.249921 (+ 1318us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.250015 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":93,"threads_started":1}
I20260810 18:11:54.252518 2847 tablet_service.cc:1467] Processing CreateTablet for tablet d938deaa820f441ebffeea91e4d404fb (DEFAULT_TABLE table=test-table [id=930dc70b00f24179a08c00ab567cf13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:54.252945 3076 tablet_service.cc:1467] Processing CreateTablet for tablet d938deaa820f441ebffeea91e4d404fb (DEFAULT_TABLE table=test-table [id=930dc70b00f24179a08c00ab567cf13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:54.253435 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.240803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:40235 (local address 127.0.0.1:40656)
0810 18:11:54.241044 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:54.241058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:54.241149 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:54.246160 (+ 5011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:54.246163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:54.246180 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:54.246253 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.246258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.250550 (+ 4292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.250553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.251837 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.251849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.253049 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.253053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.253142 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.253166 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.253255 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.253326 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260810 18:11:54.253585 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.241086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.23.65.67:40235)
0810 18:11:54.245724 (+ 4638us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.245736 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.245759 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:11:54.245831 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.245836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.246010 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.246385 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:54.246528 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.250417 (+ 3889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.252042 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.252916 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.253031 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.253184 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.253285 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.253390 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4522,"thread_start_us":191,"threads_started":1}
I20260810 18:11:54.253796 2847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d938deaa820f441ebffeea91e4d404fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:54.253974 3076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d938deaa820f441ebffeea91e4d404fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:54.256387 3279 tablet_service.cc:1467] Processing CreateTablet for tablet d938deaa820f441ebffeea91e4d404fb (DEFAULT_TABLE table=test-table [id=930dc70b00f24179a08c00ab567cf13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:54.257381 3279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d938deaa820f441ebffeea91e4d404fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:54.261097 3421 tablet_bootstrap.cc:492] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: Bootstrap starting.
I20260810 18:11:54.264200 3421 tablet_bootstrap.cc:654] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:54.265628 3421 log.cc:824] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:54.266381 3425 tablet_bootstrap.cc:492] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: Bootstrap starting.
I20260810 18:11:54.266631 3422 tablet_bootstrap.cc:492] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: Bootstrap starting.
I20260810 18:11:54.267853 3421 tablet_bootstrap.cc:492] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: No bootstrap required, opened a new log
I20260810 18:11:54.268069 3421 ts_tablet_manager.cc:1397] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:11:54.269129 3425 tablet_bootstrap.cc:654] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:54.269292 3422 tablet_bootstrap.cc:654] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:54.270448 3425 log.cc:824] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:54.270696 3422 log.cc:824] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:54.272698 3422 tablet_bootstrap.cc:492] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: No bootstrap required, opened a new log
I20260810 18:11:54.272884 3422 ts_tablet_manager.cc:1397] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:11:54.272675 3421 raft_consensus.cc:348] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } }
I20260810 18:11:54.272951 3421 raft_consensus.cc:374] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:54.273008 3425 tablet_bootstrap.cc:492] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: No bootstrap required, opened a new log
I20260810 18:11:54.273016 3421 raft_consensus.cc:729] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d63c94c94d4c77995cbf795453d93e, State: Initialized, Role: FOLLOWER
I20260810 18:11:54.273187 3425 ts_tablet_manager.cc:1397] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 18:11:54.273512 3421 consensus_queue.cc:260] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } }
I20260810 18:11:54.274729 3421 ts_tablet_manager.cc:1428] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260810 18:11:54.279932 3168 heartbeater.cc:503] Master 127.23.65.126:46139 was elected leader, sending a full tablet report...
I20260810 18:11:54.279884 3425 raft_consensus.cc:348] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } }
I20260810 18:11:54.280151 3425 raft_consensus.cc:374] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:54.280210 3425 raft_consensus.cc:729] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e697639a4d4f519a9fa399614b49b8, State: Initialized, Role: FOLLOWER
I20260810 18:11:54.279368 3422 raft_consensus.cc:348] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } }
I20260810 18:11:54.280882 3422 raft_consensus.cc:374] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:54.280937 3422 raft_consensus.cc:729] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92666cfe2574cff8a51c370ad089327, State: Initialized, Role: FOLLOWER
I20260810 18:11:54.281378 3422 consensus_queue.cc:260] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } }
I20260810 18:11:54.281785 3425 consensus_queue.cc:260] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } }
I20260810 18:11:54.282620 3422 ts_tablet_manager.cc:1428] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260810 18:11:54.282819 3425 ts_tablet_manager.cc:1428] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20260810 18:11:54.282968 2960 heartbeater.cc:503] Master 127.23.65.126:46139 was elected leader, sending a full tablet report...
I20260810 18:11:54.287092 3370 heartbeater.cc:503] Master 127.23.65.126:46139 was elected leader, sending a full tablet report...
I20260810 18:11:54.298923 3431 raft_consensus.cc:482] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:54.299098 3431 raft_consensus.cc:504] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } }
I20260810 18:11:54.300209 3431 leader_election.cc:288] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92666cfe2574cff8a51c370ad089327 (127.23.65.65:34559), 70d63c94c94d4c77995cbf795453d93e (127.23.65.66:40939)
I20260810 18:11:54.303776 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.300537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:37381 (local address 127.23.65.66:40939)
0810 18:11:54.300711 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.300715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.300726 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:11:54.300769 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.300772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.300808 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.300870 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:54.301125 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.301711 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.302868 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.303245 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.303346 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.303400 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.303580 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.303644 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:11:54.303808 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.300396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:40939 (local address 127.23.65.67:37381)
0810 18:11:54.300498 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:54.300512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:54.300587 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:54.300893 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:54.300896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:54.300913 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:54.301027 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.301033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.301817 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.301820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.302752 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.302764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.303346 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.303350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.303452 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.303474 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.303574 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.303644 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260810 18:11:54.303845 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.300495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:37115 (local address 127.23.65.65:34559)
0810 18:11:54.300611 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.300616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.300777 (+ 161us) server_negotiation.cc:407] Connection header received
0810 18:11:54.300901 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.300903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.300941 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.300999 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:54.301263 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.301742 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.302857 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.303239 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.303350 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.303401 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.303666 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.303736 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:11:54.303845 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.300396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:34559 (local address 127.23.65.67:37115)
0810 18:11:54.300702 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:54.300714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:54.300797 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:54.301017 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:54.301020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:54.301037 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:54.301162 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.301168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.301849 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.301853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.302734 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:54.302744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.303348 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.303352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:54.303520 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.303568 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 18:11:54.303638 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:11:54.303708 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":143}
I20260810 18:11:54.304347 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d938deaa820f441ebffeea91e4d404fb" candidate_uuid: "56e697639a4d4f519a9fa399614b49b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d63c94c94d4c77995cbf795453d93e" is_pre_election: true
I20260810 18:11:54.304610 3099 raft_consensus.cc:2454] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e697639a4d4f519a9fa399614b49b8 in term 0.
I20260810 18:11:54.304684 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d938deaa820f441ebffeea91e4d404fb" candidate_uuid: "56e697639a4d4f519a9fa399614b49b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92666cfe2574cff8a51c370ad089327" is_pre_election: true
I20260810 18:11:54.304970 2882 raft_consensus.cc:2454] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e697639a4d4f519a9fa399614b49b8 in term 0.
I20260810 18:11:54.305050 3199 leader_election.cc:302] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e697639a4d4f519a9fa399614b49b8, 70d63c94c94d4c77995cbf795453d93e; no voters:
I20260810 18:11:54.305455 3431 raft_consensus.cc:2781] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:54.305541 3431 raft_consensus.cc:482] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:54.305572 3431 raft_consensus.cc:3037] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:54.307370 3431 raft_consensus.cc:504] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } }
I20260810 18:11:54.307760 3431 leader_election.cc:288] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [CANDIDATE]: Term 1 election: Requested vote from peers e92666cfe2574cff8a51c370ad089327 (127.23.65.65:34559), 70d63c94c94d4c77995cbf795453d93e (127.23.65.66:40939)
I20260810 18:11:54.308110 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d938deaa820f441ebffeea91e4d404fb" candidate_uuid: "56e697639a4d4f519a9fa399614b49b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92666cfe2574cff8a51c370ad089327"
I20260810 18:11:54.308220 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d938deaa820f441ebffeea91e4d404fb" candidate_uuid: "56e697639a4d4f519a9fa399614b49b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d63c94c94d4c77995cbf795453d93e"
I20260810 18:11:54.308239 2882 raft_consensus.cc:3037] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:54.308351 3099 raft_consensus.cc:3037] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:54.309623 2882 raft_consensus.cc:2454] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e697639a4d4f519a9fa399614b49b8 in term 1.
I20260810 18:11:54.309918 3197 leader_election.cc:302] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e697639a4d4f519a9fa399614b49b8, e92666cfe2574cff8a51c370ad089327; no voters:
I20260810 18:11:54.310122 3431 raft_consensus.cc:2781] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:54.310370 3099 raft_consensus.cc:2454] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e697639a4d4f519a9fa399614b49b8 in term 1.
I20260810 18:11:54.310477 3431 raft_consensus.cc:686] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 1 LEADER]: Becoming Leader. State: Replica: 56e697639a4d4f519a9fa399614b49b8, State: Running, Role: LEADER
I20260810 18:11:54.310827 3431 consensus_queue.cc:237] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } }
I20260810 18:11:54.314385 1466 catalog_manager.cc:5495] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e697639a4d4f519a9fa399614b49b8 (127.23.65.67). New cstate: current_term: 1 leader_uuid: "56e697639a4d4f519a9fa399614b49b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } health_report { overall_health: UNKNOWN } } }
I20260810 18:11:54.330515 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.327184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.23.65.67:40235)
0810 18:11:54.327327 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.327330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.327344 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:54.327472 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.327475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.327549 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.327647 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:54.327868 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.328414 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.329230 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.329506 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.329595 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.329817 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:54.330210 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:54.330307 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.330332 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.330380 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:11:54.336923 2882 raft_consensus.cc:1261] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [term 1 FOLLOWER]: Refusing update from remote peer 56e697639a4d4f519a9fa399614b49b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:54.336925 3099 raft_consensus.cc:1261] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [term 1 FOLLOWER]: Refusing update from remote peer 56e697639a4d4f519a9fa399614b49b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:54.337663 3433 consensus_queue.cc:1035] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:54.337857 3431 consensus_queue.cc:1035] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:54.346957 3437 mvcc.cc:205] Tried to move back new op lower bound from 7317035066716884992 to 7317035066619887616. Current Snapshot: MvccSnapshot[applied={T|T < 7317035066716884992}]
I20260810 18:11:54.347317 3439 mvcc.cc:205] Tried to move back new op lower bound from 7317035066716884992 to 7317035066619887616. Current Snapshot: MvccSnapshot[applied={T|T < 7317035066716884992}]
I20260810 18:11:54.348336 3440 mvcc.cc:205] Tried to move back new op lower bound from 7317035066716884992 to 7317035066619887616. Current Snapshot: MvccSnapshot[applied={T|T < 7317035066716884992}]
I20260810 18:11:54.357678 1466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:54.357836 1466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:54.362421 1466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b729eb5df9e496cbbfeaa1fc28ed39e: Sending DeleteTablet for 3 replicas of tablet d938deaa820f441ebffeea91e4d404fb
I20260810 18:11:54.363330 3279 tablet_service.cc:1514] Processing DeleteTablet for tablet d938deaa820f441ebffeea91e4d404fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:54 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:40656
I20260810 18:11:54.363380 3076 tablet_service.cc:1514] Processing DeleteTablet for tablet d938deaa820f441ebffeea91e4d404fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:54 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:52748
I20260810 18:11:54.364086 3456 tablet_replica.cc:330] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: stopping tablet replica
I20260810 18:11:54.364279 2847 tablet_service.cc:1514] Processing DeleteTablet for tablet d938deaa820f441ebffeea91e4d404fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:54 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:55208
I20260810 18:11:54.364424 3456 raft_consensus.cc:2229] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:54.364513 3454 tablet_replica.cc:330] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: stopping tablet replica
I20260810 18:11:54.364854 3456 raft_consensus.cc:2258] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:54.364974 3454 raft_consensus.cc:2229] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:54.365406 3454 raft_consensus.cc:2258] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:54.365746 3458 tablet_replica.cc:330] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: stopping tablet replica
I20260810 18:11:54.366045 3458 raft_consensus.cc:2229] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:54.366426 3458 raft_consensus.cc:2258] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:54.367424 3454 ts_tablet_manager.cc:1905] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:54.367672 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 1424
I20260810 18:11:54.370440 3456 ts_tablet_manager.cc:1905] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:54.373831 3454 ts_tablet_manager.cc:1918] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:54.374029 3454 log.cc:1196] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/d938deaa820f441ebffeea91e4d404fb
I20260810 18:11:54.374471 3454 ts_tablet_manager.cc:1939] T d938deaa820f441ebffeea91e4d404fb P 70d63c94c94d4c77995cbf795453d93e: Deleting consensus metadata
I20260810 18:11:54.374722 3456 ts_tablet_manager.cc:1918] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:54.374821 3456 log.cc:1196] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/d938deaa820f441ebffeea91e4d404fb
I20260810 18:11:54.375094 3456 ts_tablet_manager.cc:1939] T d938deaa820f441ebffeea91e4d404fb P 56e697639a4d4f519a9fa399614b49b8: Deleting consensus metadata
W20260810 18:11:54.382894 3000 connection.cc:535] server connection from 127.0.0.1:52748 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 18:11:54.383239 3200 connection.cc:535] server connection from 127.0.0.1:40656 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 18:11:54.385499 3458 ts_tablet_manager.cc:1905] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:54.385576 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.366360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.23.65.125:38417)
0810 18:11:54.366502 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.366507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.376368 (+ 9861us) server_negotiation.cc:407] Connection header received
0810 18:11:54.376470 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.376473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.376535 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.376630 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.377593 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.378440 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.379585 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.379960 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.380064 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.381911 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.381916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.381919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.381946 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.383672 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.384148 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.384152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.384154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.384253 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.384570 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.384573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.384575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.384819 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.384924 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.385300 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.385421 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:11:54.388967 3458 ts_tablet_manager.cc:1918] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:54.389143 3458 log.cc:1196] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/d938deaa820f441ebffeea91e4d404fb
I20260810 18:11:54.389688 3458 ts_tablet_manager.cc:1939] T d938deaa820f441ebffeea91e4d404fb P e92666cfe2574cff8a51c370ad089327: Deleting consensus metadata
W20260810 18:11:54.390939 2773 connection.cc:441] server connection from 127.0.0.1:55208 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55208 (request call id 1) could send its response
I20260810 18:11:54.395802 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.366468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.23.65.124:36711)
0810 18:11:54.366592 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.366596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.386162 (+ 19566us) server_negotiation.cc:407] Connection header received
0810 18:11:54.386358 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.386362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.386422 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.386480 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.387373 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.388104 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.389476 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.389831 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.389908 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.391780 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.391784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.391787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.391814 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.393489 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.394057 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.394061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.394063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.394174 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.394477 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.394481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.394483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.394968 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.395083 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.395475 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.395637 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:11:54.399158 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.388530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.23.65.125:38417)
0810 18:11:54.388647 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.388651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.389872 (+ 1221us) server_negotiation.cc:407] Connection header received
0810 18:11:54.390051 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.390054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.390096 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.390144 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.391711 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.392436 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.393695 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.394022 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.394094 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.395694 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.395698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.395701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.395728 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.397263 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.397740 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.397743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.397745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.397840 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.398154 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.398178 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.398179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.398387 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.398504 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.398817 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.398992 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:11:54.401248 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:54.388992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.23.65.124:36711)
0810 18:11:54.389245 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:54.389250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:54.391445 (+ 2195us) server_negotiation.cc:407] Connection header received
0810 18:11:54.391637 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:54.391641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:54.391686 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:54.391734 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:54.392623 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.393358 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.395136 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:54.395470 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:54.395590 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:54.397533 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:54.397537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:54.397540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:54.397565 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:54.399249 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.399970 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.399974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.399976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.400056 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:54.400323 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:54.400326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:54.400328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:54.400495 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:54.400607 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:54.400839 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 18:11:54.400975 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:11:55.352309 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.351765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.66:32895)
0810 18:11:55.352070 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.352185 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
W20260810 18:11:55.352807 3168 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:46139 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
I20260810 18:11:55.353201 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.352746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.67:57765)
0810 18:11:55.352991 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.353088 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260810 18:11:55.353358 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.352959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.65:60889)
0810 18:11:55.353177 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.353254 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
W20260810 18:11:55.353456 3370 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:46139 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
W20260810 18:11:55.353718 2960 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:46139 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
I20260810 18:11:55.862689 3482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b729eb5df9e496cbbfeaa1fc28ed39e)
I20260810 18:11:55.862903 3482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } }
I20260810 18:11:55.863377 3484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b729eb5df9e496cbbfeaa1fc28ed39e)
I20260810 18:11:55.863605 3484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } }
I20260810 18:11:55.863920 3482 leader_election.cc:288] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b729eb5df9e496cbbfeaa1fc28ed39e (127.23.65.126:46139), 510ce43378724dd8816f3b2dd123516f (127.23.65.125:38417)
I20260810 18:11:55.864655 3484 leader_election.cc:288] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b729eb5df9e496cbbfeaa1fc28ed39e (127.23.65.126:46139), 32ee363451724bd89da561e507770d1c (127.23.65.124:36711)
I20260810 18:11:55.865136 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.864710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:45032)
0810 18:11:55.864958 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.865035 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":58,"threads_started":1}
W20260810 18:11:55.865320 2377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
I20260810 18:11:55.865396 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.864127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:45030)
0810 18:11:55.865225 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.865300 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":71,"threads_started":1}
W20260810 18:11:55.865564 2652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
I20260810 18:11:55.866268 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.866017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:45040)
0810 18:11:55.866145 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.866203 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 18:11:55.866401 2377 leader_election.cc:334] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b729eb5df9e496cbbfeaa1fc28ed39e (127.23.65.126:46139): Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
I20260810 18:11:55.866490 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.866254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:45056)
0810 18:11:55.866380 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.866426 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260810 18:11:55.866637 2652 leader_election.cc:334] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b729eb5df9e496cbbfeaa1fc28ed39e (127.23.65.126:46139): Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
I20260810 18:11:55.868582 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.864209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.23.65.125:38417)
0810 18:11:55.864517 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:55.864521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:55.864535 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:55.864688 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:55.864691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:55.864739 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:55.864813 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:55.865205 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.865938 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.867568 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.868097 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.868193 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.868254 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:55.868410 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 18:11:55.868480 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20260810 18:11:55.868580 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.864150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:38417 (local address 127.0.0.1:59702)
0810 18:11:55.864421 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.864438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:55.864549 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:55.864943 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:55.864947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:55.864969 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:55.865076 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.865082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.866061 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.866065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.867383 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.867391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.868213 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.868217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.868303 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.868323 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:11:55.868388 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:11:55.868458 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":51,"threads_started":1}
I20260810 18:11:55.869065 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ee363451724bd89da561e507770d1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "510ce43378724dd8816f3b2dd123516f" is_pre_election: true
I20260810 18:11:55.869293 2417 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ee363451724bd89da561e507770d1c in term 1.
I20260810 18:11:55.869623 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.864814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:36711 (local address 127.0.0.1:53682)
0810 18:11:55.865089 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.865116 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:55.865319 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:55.865935 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:55.865938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:55.865959 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:55.866126 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.866135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.867515 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.867519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.868450 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.868460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.869246 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.869250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.869337 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.869355 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:11:55.869427 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:11:55.869489 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":37,"thread_start_us":63,"threads_started":1}
I20260810 18:11:55.869664 2658 leader_election.cc:302] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32ee363451724bd89da561e507770d1c, 510ce43378724dd8816f3b2dd123516f; no voters: 2b729eb5df9e496cbbfeaa1fc28ed39e
I20260810 18:11:55.869895 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.865024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.23.65.124:36711)
0810 18:11:55.865699 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:55.865703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:55.865717 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:55.865758 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:55.865761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:55.865806 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:55.866237 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:55.866446 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.867383 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.868571 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.869143 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.869251 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.869305 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:55.869539 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 18:11:55.869627 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":61,"threads_started":1}
I20260810 18:11:55.869896 3482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:11:55.870007 3482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2b729eb5df9e496cbbfeaa1fc28ed39e)
I20260810 18:11:55.870059 3482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:11:55.870260 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "510ce43378724dd8816f3b2dd123516f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "32ee363451724bd89da561e507770d1c" is_pre_election: true
I20260810 18:11:55.871106 3482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } }
I20260810 18:11:55.871377 2692 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 510ce43378724dd8816f3b2dd123516f in current term 2: Already voted for candidate 32ee363451724bd89da561e507770d1c in this term.
I20260810 18:11:55.871434 3482 leader_election.cc:288] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [CANDIDATE]: Term 2 election: Requested vote from peers 2b729eb5df9e496cbbfeaa1fc28ed39e (127.23.65.126:46139), 510ce43378724dd8816f3b2dd123516f (127.23.65.125:38417)
I20260810 18:11:55.871933 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.871588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:45070)
0810 18:11:55.871788 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.871864 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260810 18:11:55.872189 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32ee363451724bd89da561e507770d1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "510ce43378724dd8816f3b2dd123516f"
I20260810 18:11:55.872313 2417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:11:55.872551 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.872347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:45084)
0810 18:11:55.872441 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.872494 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 18:11:55.872687 2652 leader_election.cc:334] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2b729eb5df9e496cbbfeaa1fc28ed39e (127.23.65.126:46139): Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
I20260810 18:11:55.873378 2417 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ee363451724bd89da561e507770d1c in term 2.
I20260810 18:11:55.873623 2658 leader_election.cc:302] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32ee363451724bd89da561e507770d1c, 510ce43378724dd8816f3b2dd123516f; no voters: 2b729eb5df9e496cbbfeaa1fc28ed39e
I20260810 18:11:55.873654 2381 leader_election.cc:302] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 510ce43378724dd8816f3b2dd123516f; no voters: 2b729eb5df9e496cbbfeaa1fc28ed39e, 32ee363451724bd89da561e507770d1c
I20260810 18:11:55.873840 3482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:11:55.873840 3484 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:11:55.874046 3482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [term 2 LEADER]: Becoming Leader. State: Replica: 32ee363451724bd89da561e507770d1c, State: Running, Role: LEADER
I20260810 18:11:55.874281 3482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } }
I20260810 18:11:55.875411 3499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ee363451724bd89da561e507770d1c. Latest consensus state: current_term: 2 leader_uuid: "32ee363451724bd89da561e507770d1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:55.875510 3499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:55.875825 3503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:55.878201 3503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:55.878584 3503 catalog_manager.cc:1259] Loaded cluster ID: 832069d7906941fa973f3b2ee8c3eb56
I20260810 18:11:55.878633 3503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:55.879168 3503 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:55.879576 3503 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Loaded TSK: 0
I20260810 18:11:55.880241 3503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:11:55.904667 2677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:55.907806 2677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:55.910630 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.910383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:45116)
0810 18:11:55.910488 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.910546 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 18:11:55.910774 2417 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [term 2 FOLLOWER]: Refusing update from remote peer 32ee363451724bd89da561e507770d1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 18:11:55.911268 3482 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 18:11:55.911358 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.911109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.0.0.1:45130)
0810 18:11:55.911214 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.911272 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 18:11:55.911494 2652 consensus_peers.cc:487] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c -> Peer 2b729eb5df9e496cbbfeaa1fc28ed39e (127.23.65.126:46139): Couldn't send request to peer 2b729eb5df9e496cbbfeaa1fc28ed39e. Status: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:11:55.912729 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ee363451724bd89da561e507770d1c. Latest consensus state: current_term: 2 leader_uuid: "32ee363451724bd89da561e507770d1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:55.912846 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:55.913889 3482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "32ee363451724bd89da561e507770d1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:55.914016 3482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:55.914466 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "32ee363451724bd89da561e507770d1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2b729eb5df9e496cbbfeaa1fc28ed39e" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46139 } } peers { permanent_uuid: "510ce43378724dd8816f3b2dd123516f" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 38417 } attrs { promote: false } } peers { permanent_uuid: "32ee363451724bd89da561e507770d1c" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 36711 } attrs { promote: false } } }
I20260810 18:11:55.914592 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510ce43378724dd8816f3b2dd123516f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:55.924420 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.920334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.23.65.67:40235)
0810 18:11:55.920569 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:55.920573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:55.920589 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:11:55.920642 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:55.920645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:55.920693 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:55.920818 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:55.921061 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.921705 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.923291 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.923887 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.923984 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.924063 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:55.924204 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 18:11:55.924283 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260810 18:11:55.924418 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.920078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:40235 (local address 127.0.0.1:40668)
0810 18:11:55.920163 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.920177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:55.920256 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:55.920838 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:55.920841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:55.920858 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:55.920956 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.920962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.921804 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.921807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.923158 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.923166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.923997 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.924000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.924086 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.924110 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:11:55.924178 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:11:55.924253 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":393}
I20260810 18:11:55.924677 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.919660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:40939 (local address 127.0.0.1:52750)
0810 18:11:55.919764 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.919780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:55.919869 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:55.921154 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:55.921157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:55.921175 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:55.921271 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.921277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.922328 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.922332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.923469 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.923479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.924353 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.924357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.924444 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.924465 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 18:11:55.924512 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 18:11:55.924582 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":53}
I20260810 18:11:55.925001 3279 tablet_service.cc:1467] Processing CreateTablet for tablet 9e95dd7e4245425f8dc367280774056b (DEFAULT_TABLE table=test-table [id=334a19f78d3a454fa7889365b05a32af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:55.925299 3279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e95dd7e4245425f8dc367280774056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:55.925308 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.920180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:34559 (local address 127.0.0.1:55220)
0810 18:11:55.920647 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.920666 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:55.920740 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:55.921117 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:55.921120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:55.921137 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:55.921254 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.921261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.922328 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.922332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.923381 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:55.923391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.924324 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.924328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:55.924998 (+ 670us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.925024 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:11:55.925091 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 18:11:55.925168 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":643,"thread_start_us":98,"threads_started":1}
I20260810 18:11:55.925343 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.920617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.23.65.65:34559)
0810 18:11:55.920879 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:55.920883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:55.920896 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:55.920947 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:55.920950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:55.920995 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:55.921350 (+ 355us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:55.921463 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.922196 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.923495 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.924181 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.924432 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.924501 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:55.925136 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 18:11:55.925225 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260810 18:11:55.925736 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.920172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.23.65.66:40939)
0810 18:11:55.920941 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:55.920945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:55.920958 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:55.921003 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:55.921007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:55.921053 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:55.921127 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:55.921385 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.922194 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.923622 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:55.924241 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:55.925483 (+ 1242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:55.925541 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:55.925572 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 18:11:55.925633 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":71,"threads_started":1}
I20260810 18:11:55.926200 3076 tablet_service.cc:1467] Processing CreateTablet for tablet 9e95dd7e4245425f8dc367280774056b (DEFAULT_TABLE table=test-table [id=334a19f78d3a454fa7889365b05a32af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:55.926427 3076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e95dd7e4245425f8dc367280774056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:55.927647 3516 tablet_bootstrap.cc:492] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: Bootstrap starting.
I20260810 18:11:55.928455 2847 tablet_service.cc:1467] Processing CreateTablet for tablet 9e95dd7e4245425f8dc367280774056b (DEFAULT_TABLE table=test-table [id=334a19f78d3a454fa7889365b05a32af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:55.928697 3516 tablet_bootstrap.cc:654] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:55.928735 2847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e95dd7e4245425f8dc367280774056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:55.929986 3516 tablet_bootstrap.cc:492] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: No bootstrap required, opened a new log
I20260810 18:11:55.930085 3516 ts_tablet_manager.cc:1397] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:55.930513 3516 raft_consensus.cc:348] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } }
I20260810 18:11:55.930640 3516 raft_consensus.cc:374] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:55.930682 3516 raft_consensus.cc:729] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e697639a4d4f519a9fa399614b49b8, State: Initialized, Role: FOLLOWER
I20260810 18:11:55.930814 3516 consensus_queue.cc:260] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } }
I20260810 18:11:55.931141 3516 ts_tablet_manager.cc:1428] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:55.931146 3519 tablet_bootstrap.cc:492] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327: Bootstrap starting.
I20260810 18:11:55.931648 3517 tablet_bootstrap.cc:492] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: Bootstrap starting.
I20260810 18:11:55.932614 3519 tablet_bootstrap.cc:654] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:55.933631 3369 heartbeater.cc:503] Master 127.23.65.124:36711 was elected leader, sending a full tablet report...
I20260810 18:11:55.933840 3519 tablet_bootstrap.cc:492] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327: No bootstrap required, opened a new log
I20260810 18:11:55.933979 3519 ts_tablet_manager.cc:1397] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:11:55.934478 3517 tablet_bootstrap.cc:654] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:55.934466 3519 raft_consensus.cc:348] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } }
I20260810 18:11:55.934551 3519 raft_consensus.cc:374] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:55.934587 3519 raft_consensus.cc:729] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92666cfe2574cff8a51c370ad089327, State: Initialized, Role: FOLLOWER
I20260810 18:11:55.934724 3519 consensus_queue.cc:260] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } }
I20260810 18:11:55.935042 3519 ts_tablet_manager.cc:1428] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:55.935997 3517 tablet_bootstrap.cc:492] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: No bootstrap required, opened a new log
I20260810 18:11:55.936090 3517 ts_tablet_manager.cc:1397] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 18:11:55.936281 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.935400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.65:44175)
0810 18:11:55.936148 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.936202 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":90,"threads_started":1}
I20260810 18:11:55.936501 3517 raft_consensus.cc:348] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } }
I20260810 18:11:55.936600 2959 heartbeater.cc:503] Master 127.23.65.124:36711 was elected leader, sending a full tablet report...
I20260810 18:11:55.936620 3517 raft_consensus.cc:374] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:55.936653 3517 raft_consensus.cc:729] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d63c94c94d4c77995cbf795453d93e, State: Initialized, Role: FOLLOWER
I20260810 18:11:55.936770 3517 consensus_queue.cc:260] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } }
I20260810 18:11:55.936942 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.936726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.65:32769)
0810 18:11:55.936832 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.936879 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 18:11:55.937119 3517 ts_tablet_manager.cc:1428] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:55.938548 3166 heartbeater.cc:503] Master 127.23.65.124:36711 was elected leader, sending a full tablet report...
I20260810 18:11:55.938721 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.932044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.67:33897)
0810 18:11:55.935126 (+ 3082us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.935191 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3011,"thread_start_us":98,"threads_started":1}
I20260810 18:11:55.939563 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.939130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.67:47321)
0810 18:11:55.939394 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.939451 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
I20260810 18:11:55.939792 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.937494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.66:39695)
0810 18:11:55.939500 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.939580 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1939,"thread_start_us":92,"threads_started":1}
I20260810 18:11:55.940349 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:55.940168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.66:54537)
0810 18:11:55.940253 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:55.940299 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 18:11:55.947654 3372 tablet.cc:2369] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:11:55.987834 3170 tablet.cc:2369] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:55.996690 3520 raft_consensus.cc:482] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:55.996858 3520 raft_consensus.cc:504] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } }
I20260810 18:11:55.997328 3520 leader_election.cc:288] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70d63c94c94d4c77995cbf795453d93e (127.23.65.66:40939), e92666cfe2574cff8a51c370ad089327 (127.23.65.65:34559)
I20260810 18:11:55.997776 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e95dd7e4245425f8dc367280774056b" candidate_uuid: "56e697639a4d4f519a9fa399614b49b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d63c94c94d4c77995cbf795453d93e" is_pre_election: true
I20260810 18:11:55.997800 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e95dd7e4245425f8dc367280774056b" candidate_uuid: "56e697639a4d4f519a9fa399614b49b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92666cfe2574cff8a51c370ad089327" is_pre_election: true
I20260810 18:11:55.997947 3099 raft_consensus.cc:2454] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e697639a4d4f519a9fa399614b49b8 in term 0.
I20260810 18:11:55.997982 2882 raft_consensus.cc:2454] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e697639a4d4f519a9fa399614b49b8 in term 0.
I20260810 18:11:55.998265 3199 leader_election.cc:302] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e697639a4d4f519a9fa399614b49b8, 70d63c94c94d4c77995cbf795453d93e; no voters:
I20260810 18:11:55.998462 3520 raft_consensus.cc:2781] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:55.998529 3520 raft_consensus.cc:482] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:55.998559 3520 raft_consensus.cc:3037] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:55.999627 3520 raft_consensus.cc:504] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } }
I20260810 18:11:55.999997 3520 leader_election.cc:288] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [CANDIDATE]: Term 1 election: Requested vote from peers 70d63c94c94d4c77995cbf795453d93e (127.23.65.66:40939), e92666cfe2574cff8a51c370ad089327 (127.23.65.65:34559)
I20260810 18:11:56.000352 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e95dd7e4245425f8dc367280774056b" candidate_uuid: "56e697639a4d4f519a9fa399614b49b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d63c94c94d4c77995cbf795453d93e"
I20260810 18:11:56.000408 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e95dd7e4245425f8dc367280774056b" candidate_uuid: "56e697639a4d4f519a9fa399614b49b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92666cfe2574cff8a51c370ad089327"
I20260810 18:11:56.000463 3099 raft_consensus.cc:3037] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:56.000530 2882 raft_consensus.cc:3037] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:56.001370 3099 raft_consensus.cc:2454] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e697639a4d4f519a9fa399614b49b8 in term 1.
I20260810 18:11:56.001643 3199 leader_election.cc:302] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e697639a4d4f519a9fa399614b49b8, 70d63c94c94d4c77995cbf795453d93e; no voters:
I20260810 18:11:56.001659 2882 raft_consensus.cc:2454] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e697639a4d4f519a9fa399614b49b8 in term 1.
I20260810 18:11:56.001842 3520 raft_consensus.cc:2781] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:56.002094 3520 raft_consensus.cc:686] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 1 LEADER]: Becoming Leader. State: Replica: 56e697639a4d4f519a9fa399614b49b8, State: Running, Role: LEADER
I20260810 18:11:56.002249 3520 consensus_queue.cc:237] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } }
I20260810 18:11:56.002817 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:56.002537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.67:50803)
0810 18:11:56.002649 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:56.002714 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 18:11:56.004014 2675 catalog_manager.cc:5495] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e697639a4d4f519a9fa399614b49b8 (127.23.65.67). New cstate: current_term: 1 leader_uuid: "56e697639a4d4f519a9fa399614b49b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e697639a4d4f519a9fa399614b49b8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 40235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 } health_report { overall_health: UNKNOWN } } }
W20260810 18:11:56.011677 2963 tablet.cc:2369] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:56.025178 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:56.022102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.23.65.67:40235)
0810 18:11:56.022245 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:56.022249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:56.022311 (+ 62us) server_negotiation.cc:407] Connection header received
0810 18:11:56.022436 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:56.022439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:56.022485 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:56.022545 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:56.022787 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:56.023306 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:56.024188 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:56.024412 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:56.024498 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:56.024746 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:56.024827 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:56.024885 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:56.024987 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 18:11:56.025040 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:11:56.027431 3099 raft_consensus.cc:1261] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [term 1 FOLLOWER]: Refusing update from remote peer 56e697639a4d4f519a9fa399614b49b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:56.027431 2882 raft_consensus.cc:1261] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [term 1 FOLLOWER]: Refusing update from remote peer 56e697639a4d4f519a9fa399614b49b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:56.027930 3520 consensus_queue.cc:1035] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92666cfe2574cff8a51c370ad089327" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:56.028110 3530 consensus_queue.cc:1035] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d63c94c94d4c77995cbf795453d93e" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:56.029773 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:56.029455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.65:49041)
0810 18:11:56.029626 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:56.029689 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 18:11:56.030011 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:56.029742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.66:58273)
0810 18:11:56.029874 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:56.029936 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 18:11:56.032809 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:56.032560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.67:53921)
0810 18:11:56.032671 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:56.032730 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 18:11:56.034787 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:56.034509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.65:58235)
0810 18:11:56.034656 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:56.034710 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 18:11:56.035056 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:56.034779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.66:40105)
0810 18:11:56.034927 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:56.034982 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 18:11:56.037096 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:56.034135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46139 (local address 127.23.65.67:48597)
0810 18:11:56.036961 (+ 2826us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:56.037020 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2756}
I20260810 18:11:56.039156 2675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:56.039296 2675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:56.042682 2675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32ee363451724bd89da561e507770d1c: Sending DeleteTablet for 3 replicas of tablet 9e95dd7e4245425f8dc367280774056b
I20260810 18:11:56.043378 3076 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e95dd7e4245425f8dc367280774056b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:56 UTC) from {username='oryx', principal='oryx/127.23.65.124@KRBTEST.COM'} at 127.0.0.1:52750
I20260810 18:11:56.043444 3279 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e95dd7e4245425f8dc367280774056b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:56 UTC) from {username='oryx', principal='oryx/127.23.65.124@KRBTEST.COM'} at 127.0.0.1:40668
I20260810 18:11:56.043488 2847 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e95dd7e4245425f8dc367280774056b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:56 UTC) from {username='oryx', principal='oryx/127.23.65.124@KRBTEST.COM'} at 127.0.0.1:55220
I20260810 18:11:56.043787 3550 tablet_replica.cc:330] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: stopping tablet replica
I20260810 18:11:56.043888 3551 tablet_replica.cc:330] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: stopping tablet replica
I20260810 18:11:56.043939 3550 raft_consensus.cc:2229] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:56.044019 3551 raft_consensus.cc:2229] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:56.044075 3550 raft_consensus.cc:2258] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:56.044234 3551 raft_consensus.cc:2258] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:56.044389 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 2622
I20260810 18:11:56.044488 3554 tablet_replica.cc:330] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327: stopping tablet replica
I20260810 18:11:56.044597 3554 raft_consensus.cc:2229] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:56.044699 3554 raft_consensus.cc:2258] T 9e95dd7e4245425f8dc367280774056b P e92666cfe2574cff8a51c370ad089327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:56.044941 3550 ts_tablet_manager.cc:1905] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:56.046144 3551 ts_tablet_manager.cc:1905] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:56.048270 3550 ts_tablet_manager.cc:1918] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:56.048305 3551 ts_tablet_manager.cc:1918] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:56.048365 3551 log.cc:1196] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/9e95dd7e4245425f8dc367280774056b
I20260810 18:11:56.048364 3550 log.cc:1196] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/9e95dd7e4245425f8dc367280774056b
I20260810 18:11:56.048606 3551 ts_tablet_manager.cc:1939] T 9e95dd7e4245425f8dc367280774056b P 56e697639a4d4f519a9fa399614b49b8: Deleting consensus metadata
I20260810 18:11:56.048655 3550 ts_tablet_manager.cc:1939] T 9e95dd7e4245425f8dc367280774056b P 70d63c94c94d4c77995cbf795453d93e: Deleting consensus metadata
I20260810 18:11:56.050397 2657 catalog_manager.cc:4841] TS 70d63c94c94d4c77995cbf795453d93e (127.23.65.66:40939): tablet 9e95dd7e4245425f8dc367280774056b (table test-table [id=334a19f78d3a454fa7889365b05a32af]) successfully deleted
I20260810 18:11:56.050513 2656 catalog_manager.cc:4841] TS 56e697639a4d4f519a9fa399614b49b8 (127.23.65.67:40235): tablet 9e95dd7e4245425f8dc367280774056b (table test-table [id=334a19f78d3a454fa7889365b05a32af]) successfully deleted
I20260810 18:11:56.060075 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 2970
I20260810 18:11:56.061965 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:56.060170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:34559 (local address 127.0.0.1:55236)
0810 18:11:56.061811 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:56.061874 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1538}
W20260810 18:11:56.064666 2652 catalog_manager.cc:4584] TS e92666cfe2574cff8a51c370ad089327 (127.23.65.65:34559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e95dd7e4245425f8dc367280774056b: Network error: Client connection negotiation failed: client connection to 127.23.65.65:34559: connect: Connection refused (error 111)
I20260810 18:11:56.072016 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 3175
I20260810 18:11:56.092646 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 2182
I20260810 18:11:56.104678 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 2358
2026-08-10T18:11:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5482 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:56 dist-test-slave-jpj1 krb5kdc[3559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:56 dist-test-slave-jpj1 krb5kdc[3559](info): set up 2 sockets
Aug 10 18:11:56 dist-test-slave-jpj1 krb5kdc[3559](info): commencing operation
krb5kdc: starting...
W20260810 18:11:57.269960 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:11:57 dist-test-slave-jpj1 krb5kdc[3559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.126.keytab.
Entry for principal oryx/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.126.keytab.
I20260810 18:11:57.394542 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40549
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.126.keytab
--principal=oryx/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:40549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:57.509861 3579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:57.510143 3579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:57.510210 3579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:57.514257 3579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:57.514318 3579 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:11:57.514343 3579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:57.514365 3579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:57.514394 3579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:57.514424 3579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:57.519728 3579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:40549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.126.keytab
--principal=oryx/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:57.520828 3579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:57.521916 3579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:57.530547 3588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:57.530555 3586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:57.530629 3579 server_base.cc:1034] running on GCE node
W20260810 18:11:57.530553 3585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:57.531477 3579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:57.532598 3579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:57.533766 3579 hybrid_clock.cc:648] HybridClock initialized: now 1786385517533748 us; error 28 us; skew 500 ppm
Aug 10 18:11:57 dist-test-slave-jpj1 krb5kdc[3559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385517, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:57.536479 3579 init.cc:377] Logged in from keytab as oryx/127.23.65.126@KRBTEST.COM (short username oryx)
I20260810 18:11:57.537371 3579 webserver.cc:460] Webserver started at http://127.23.65.126:45267/ using document root <none> and password file <none>
I20260810 18:11:57.537917 3579 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:57.537985 3579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:57.538185 3579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:57.540266 3579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "3c0c64a9a1024e8cadbd97c030f2368a"
format_stamp: "Formatted at 2026-08-10 18:11:57 on dist-test-slave-jpj1"
server_key: "e88af6621d05c0bd4849be3af3225887"
server_key_iv: "57aad1ce85af123aff0b24830d557faf"
server_key_version: "encryptionkey@0"
I20260810 18:11:57.540800 3579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "3c0c64a9a1024e8cadbd97c030f2368a"
format_stamp: "Formatted at 2026-08-10 18:11:57 on dist-test-slave-jpj1"
server_key: "e88af6621d05c0bd4849be3af3225887"
server_key_iv: "57aad1ce85af123aff0b24830d557faf"
server_key_version: "encryptionkey@0"
I20260810 18:11:57.544366 3579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:11:57.546629 3596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:57.547821 3579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:11:57.547933 3579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "3c0c64a9a1024e8cadbd97c030f2368a"
format_stamp: "Formatted at 2026-08-10 18:11:57 on dist-test-slave-jpj1"
server_key: "e88af6621d05c0bd4849be3af3225887"
server_key_iv: "57aad1ce85af123aff0b24830d557faf"
server_key_version: "encryptionkey@0"
I20260810 18:11:57.548022 3579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:57.568609 3579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:57.571745 3579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:57.571936 3579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:57.579993 3579 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:40549
I20260810 18:11:57.580010 3695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:40549 every 8 connection(s)
I20260810 18:11:57.580951 3579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:57.581164 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 3579
I20260810 18:11:57.581270 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:11:57.581629 23813 external_mini_cluster.cc:1428] Setting key c2a0dc48372fea9762639410d90872ad
I20260810 18:11:57.584244 3696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:57.590754 3696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: Bootstrap starting.
Aug 10 18:11:57 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:57.593447 3696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:57.594763 3696 log.cc:824] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:57.596714 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:57.583061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.23.65.126:40549)
0810 18:11:57.583500 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:57.583521 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:57.583599 (+ 78us) server_negotiation.cc:407] Connection header received
0810 18:11:57.584474 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:57.584499 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:57.584909 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:57.585280 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:57.585928 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.587415 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.588366 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.589052 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.589232 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:57.591826 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:57.591833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:57.591851 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:57.591886 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:57.593749 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:57.594155 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:57.594160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:57.594166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:57.594312 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:57.594527 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:57.594530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:57.594531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:57.595032 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:57.595249 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:57.595490 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 18:11:57.595797 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260810 18:11:57.597033 3696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: No bootstrap required, opened a new log
I20260810 18:11:57.600260 3696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0c64a9a1024e8cadbd97c030f2368a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40549 } }
I20260810 18:11:57.600466 3696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:57.600514 3696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0c64a9a1024e8cadbd97c030f2368a, State: Initialized, Role: FOLLOWER
I20260810 18:11:57.600867 3696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0c64a9a1024e8cadbd97c030f2368a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40549 } }
I20260810 18:11:57.600976 3696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:57.601018 3696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:57.601086 3696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:57.601862 3696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0c64a9a1024e8cadbd97c030f2368a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40549 } }
I20260810 18:11:57.602149 3696 leader_election.cc:302] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c0c64a9a1024e8cadbd97c030f2368a; no voters:
I20260810 18:11:57.602391 3696 leader_election.cc:288] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:57.602517 3705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:57.602730 3705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [term 1 LEADER]: Becoming Leader. State: Replica: 3c0c64a9a1024e8cadbd97c030f2368a, State: Running, Role: LEADER
I20260810 18:11:57.603053 3705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0c64a9a1024e8cadbd97c030f2368a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40549 } }
I20260810 18:11:57.603464 3696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:57.605075 3706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c0c64a9a1024e8cadbd97c030f2368a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0c64a9a1024e8cadbd97c030f2368a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40549 } } }
I20260810 18:11:57.605199 3706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:57.605513 3709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c0c64a9a1024e8cadbd97c030f2368a. Latest consensus state: current_term: 1 leader_uuid: "3c0c64a9a1024e8cadbd97c030f2368a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0c64a9a1024e8cadbd97c030f2368a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40549 } } }
I20260810 18:11:57.605911 3709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a [sys.catalog]: This master's current role is: LEADER
W20260810 18:11:57.608311 3730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:57.608388 3730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:57.608479 3728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:57.609182 3728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:57.614387 3728 catalog_manager.cc:1347] Generated new cluster ID: fc41247e99de4f2292ff3b84533ee5c8
I20260810 18:11:57.614454 3728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:57.620836 3728 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:57.622023 3728 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:57.630335 3728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: Generated new TSK 0
I20260810 18:11:57.631217 3728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab.
Entry for principal oryx/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab.
I20260810 18:11:57.663021 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40549
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab
--principal=oryx/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:57.776736 3742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:57.777005 3742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:57.777068 3742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:57.781052 3742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:57.781111 3742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:57.781162 3742 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:11:57.781217 3742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:11:57.786341 3742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab
--principal=oryx/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.65.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:57.787437 3742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:57.788470 3742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:57.797745 3752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:57.797752 3748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:57.797752 3749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:57.798076 3742 server_base.cc:1034] running on GCE node
I20260810 18:11:57.798447 3742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:57.799039 3742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:57.800213 3742 hybrid_clock.cc:648] HybridClock initialized: now 1786385517800190 us; error 33 us; skew 500 ppm
Aug 10 18:11:57 dist-test-slave-jpj1 krb5kdc[3559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385517, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:57.802855 3742 init.cc:377] Logged in from keytab as oryx/127.23.65.65@KRBTEST.COM (short username oryx)
I20260810 18:11:57.803718 3742 webserver.cc:460] Webserver started at http://127.23.65.65:41155/ using document root <none> and password file <none>
I20260810 18:11:57.804236 3742 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:57.804303 3742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:57.804482 3742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:57.806437 3742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "cf27092e6a0f4e0b98615b02c1d01728"
format_stamp: "Formatted at 2026-08-10 18:11:57 on dist-test-slave-jpj1"
server_key: "12f400941e6cc4a48bf3bb711c95af58"
server_key_iv: "311de59cafc200810579f1924ae6f2e5"
server_key_version: "encryptionkey@0"
I20260810 18:11:57.806951 3742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "cf27092e6a0f4e0b98615b02c1d01728"
format_stamp: "Formatted at 2026-08-10 18:11:57 on dist-test-slave-jpj1"
server_key: "12f400941e6cc4a48bf3bb711c95af58"
server_key_iv: "311de59cafc200810579f1924ae6f2e5"
server_key_version: "encryptionkey@0"
I20260810 18:11:57.810819 3742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:11:57.813373 3765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:57.814602 3742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:57.814703 3742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "cf27092e6a0f4e0b98615b02c1d01728"
format_stamp: "Formatted at 2026-08-10 18:11:57 on dist-test-slave-jpj1"
server_key: "12f400941e6cc4a48bf3bb711c95af58"
server_key_iv: "311de59cafc200810579f1924ae6f2e5"
server_key_version: "encryptionkey@0"
I20260810 18:11:57.814829 3742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:57.838510 3742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:57.841210 3742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:57.841378 3742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:57.842782 3742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:57.842830 3742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:57.842871 3742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:57.842900 3742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:11:57 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385517, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:57.853389 3742 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:46099
I20260810 18:11:57.853446 3947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:46099 every 8 connection(s)
I20260810 18:11:57.854313 3742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:11:57.856613 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:57.843606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:35531 (local address 127.23.65.126:40549)
0810 18:11:57.843771 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:57.843776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:57.844582 (+ 806us) server_negotiation.cc:407] Connection header received
0810 18:11:57.845583 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:57.845587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:57.845649 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:57.845710 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:57.847407 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.848230 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.849514 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.849886 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.850005 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:57.852943 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:57.852947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:57.852950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:57.852983 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:57.854558 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:57.855162 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:57.855165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:57.855167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:57.855277 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:57.855613 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:57.855615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:57.855617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:57.855829 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:57.855914 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:57.856375 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 18:11:57.856488 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:11:57.857414 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:57.843951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40549 (local address 127.23.65.65:35531)
0810 18:11:57.844409 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:57.844458 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:57.845337 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:57.846134 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:57.846143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:57.846598 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:57.847211 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:57.847229 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.848383 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.848387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:57.849404 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:57.849415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.850066 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.850071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:57.850286 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:57.850914 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:57.850935 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:57.852735 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:57.854680 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:57.854686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:57.854703 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:57.855053 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:57.855372 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:57.855374 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:57.855375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:57.855515 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:57.855928 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:57.855933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:57.856174 (+ 241us) client_negotiation.cc:769] Sending connection context
0810 18:11:57.856331 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 18:11:57.856592 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260810 18:11:57.858611 3949 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40549
I20260810 18:11:57.858759 3949 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:57.859256 3949 heartbeater.cc:511] Master 127.23.65.126:40549 requested a full tablet report, sending...
I20260810 18:11:57.860072 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 3742
I20260810 18:11:57.860201 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:11:57.860536 23813 external_mini_cluster.cc:1428] Setting key 38de2abe3446ee8ea1d9915b36bf8572
I20260810 18:11:57.860780 3621 ts_manager.cc:194] Registered new tserver with Master: cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65:46099)
I20260810 18:11:57.862241 3621 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:35531
I20260810 18:11:57.868229 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:57.860970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:57681 (local address 127.23.65.126:40549)
0810 18:11:57.861078 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:57.861082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:57.861326 (+ 244us) server_negotiation.cc:407] Connection header received
0810 18:11:57.861414 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:57.861418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:57.861461 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:57.861522 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:57.861954 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.862645 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.863851 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.864167 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.864267 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:57.865392 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:57.865396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:57.865399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:57.865425 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:57.866691 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:57.867151 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:57.867153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:57.867154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:57.867225 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:57.867487 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:57.867488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:57.867489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:57.867674 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:57.867750 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:57.868025 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 18:11:57.868117 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:11:57.868341 3962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:57.860878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40549 (local address 127.23.65.65:57681)
0810 18:11:57.861242 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:57.861256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:57.861345 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:57.861550 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:57.861553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:57.861738 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:57.861852 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:57.861860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.862894 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.862897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:57.863714 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:57.863726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:57.864395 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:57.864398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:57.864477 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:57.864933 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:57.864943 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:57.865243 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:57.866806 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:57.866809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:57.866811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:57.867057 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:57.867309 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:57.867312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:57.867314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:57.867407 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:57.867764 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:57.867768 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:57.867868 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 18:11:57.867989 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 18:11:57.868129 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for oryx/127.23.65.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab.
Entry for principal oryx/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab.
I20260810 18:11:57.888586 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40549
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab
--principal=oryx/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:58.000470 3967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:58.000741 3967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:58.000794 3967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:58.004750 3967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:58.004804 3967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:58.004846 3967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:11:58.004907 3967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:11:58.010077 3967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab
--principal=oryx/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.65.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:58.011135 3967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:58.012238 3967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:58.022346 3985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:58.022339 3978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:58.022396 3967 server_base.cc:1034] running on GCE node
W20260810 18:11:58.022339 3987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:58.023020 3967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:58.023713 3967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:58.024875 3967 hybrid_clock.cc:648] HybridClock initialized: now 1786385518024853 us; error 32 us; skew 500 ppm
Aug 10 18:11:58 dist-test-slave-jpj1 krb5kdc[3559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385518, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:58.027637 3967 init.cc:377] Logged in from keytab as oryx/127.23.65.66@KRBTEST.COM (short username oryx)
I20260810 18:11:58.028512 3967 webserver.cc:460] Webserver started at http://127.23.65.66:36345/ using document root <none> and password file <none>
I20260810 18:11:58.029027 3967 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:58.029084 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:58.029284 3967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:58.031308 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "fb12e0294d14498b8f62452b19bcdb14"
format_stamp: "Formatted at 2026-08-10 18:11:58 on dist-test-slave-jpj1"
server_key: "d462f1c2d285a17317f6e42d8c227ea6"
server_key_iv: "04939dcc4bb113585822e60f7d966b3e"
server_key_version: "encryptionkey@0"
I20260810 18:11:58.031889 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "fb12e0294d14498b8f62452b19bcdb14"
format_stamp: "Formatted at 2026-08-10 18:11:58 on dist-test-slave-jpj1"
server_key: "d462f1c2d285a17317f6e42d8c227ea6"
server_key_iv: "04939dcc4bb113585822e60f7d966b3e"
server_key_version: "encryptionkey@0"
I20260810 18:11:58.035663 3967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:11:58.037981 3998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:58.039155 3967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:58.039276 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "fb12e0294d14498b8f62452b19bcdb14"
format_stamp: "Formatted at 2026-08-10 18:11:58 on dist-test-slave-jpj1"
server_key: "d462f1c2d285a17317f6e42d8c227ea6"
server_key_iv: "04939dcc4bb113585822e60f7d966b3e"
server_key_version: "encryptionkey@0"
I20260810 18:11:58.039383 3967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:58.049113 3967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:58.052083 3967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:58.052254 3967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:58.053766 3967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:58.053818 3967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:58.053874 3967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:58.053902 3967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:11:58 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385518, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:58.064213 3967 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:35857
I20260810 18:11:58.064332 4179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:35857 every 8 connection(s)
I20260810 18:11:58.065263 3967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:11:58.067386 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.054557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:34497 (local address 127.23.65.126:40549)
0810 18:11:58.054712 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.054716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.055654 (+ 938us) server_negotiation.cc:407] Connection header received
0810 18:11:58.056802 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.056806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.056885 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.056986 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:58.058390 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.059233 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.060675 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.060906 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.060997 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.063757 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:58.063760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:58.063763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:58.063802 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:58.065384 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.066034 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.066038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.066040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.066135 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.066470 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.066473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.066474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.066656 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:58.066731 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.067158 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.067261 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:11:58.068197 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.054981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40549 (local address 127.23.65.66:34497)
0810 18:11:58.055463 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.055518 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.056560 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.057209 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.057216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.057588 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:58.058231 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.058250 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.059368 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.059372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.060561 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.060571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.061063 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.061067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.061236 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.061792 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:58.061809 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:58.063594 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:58.065518 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.065523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.065538 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.065911 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.066239 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.066241 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.066243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.066383 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.066739 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:58.066744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:58.066971 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.067115 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.067374 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":123,"threads_started":1}
I20260810 18:11:58.069260 4180 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40549
I20260810 18:11:58.069399 4180 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:58.069890 4180 heartbeater.cc:511] Master 127.23.65.126:40549 requested a full tablet report, sending...
I20260810 18:11:58.070943 3620 ts_manager.cc:194] Registered new tserver with Master: fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857)
I20260810 18:11:58.071746 3620 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:34497
I20260810 18:11:58.075505 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 3967
I20260810 18:11:58.075665 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:11:58.076005 23813 external_mini_cluster.cc:1428] Setting key fe48dbe8f8af8b593ddcce07a608548c
I20260810 18:11:58.078054 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.071388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:58175 (local address 127.23.65.126:40549)
0810 18:11:58.071494 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.071498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.071696 (+ 198us) server_negotiation.cc:407] Connection header received
0810 18:11:58.071878 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.071882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.071924 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.071999 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:58.072469 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.072971 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.073761 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.073998 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.074083 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.075063 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:58.075066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:58.075069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:58.075094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:58.076494 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.076910 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.076913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.076914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.076993 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.077249 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.077253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.077254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.077498 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:58.077577 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.077854 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.077944 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 18:11:58.078137 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.071320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40549 (local address 127.23.65.66:58175)
0810 18:11:58.071616 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.071630 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.071776 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.072100 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.072102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.072265 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:58.072358 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.072363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.073099 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.073102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.073653 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.073661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.074098 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.074101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.074198 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.074616 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:58.074626 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:58.074937 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:58.076595 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.076598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.076600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.076830 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.077073 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.077076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.077077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.077177 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.077600 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:58.077603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:58.077706 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.077816 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.077954 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab.
Entry for principal oryx/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab.
I20260810 18:11:58.102716 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40549
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab
--principal=oryx/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:58.218005 4197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:58.218327 4197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:58.218377 4197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:58.222453 4197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:58.222520 4197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:58.222558 4197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:11:58.222599 4197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:11:58.227766 4197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab
--principal=oryx/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.65.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:58.228791 4197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:58.229841 4197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:58.239087 4204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:58.239087 4205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:58.239082 4208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:58.239346 4197 server_base.cc:1034] running on GCE node
I20260810 18:11:58.239845 4197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:58.240425 4197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:58.241580 4197 hybrid_clock.cc:648] HybridClock initialized: now 1786385518241566 us; error 25 us; skew 500 ppm
Aug 10 18:11:58 dist-test-slave-jpj1 krb5kdc[3559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385518, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:58.244297 4197 init.cc:377] Logged in from keytab as oryx/127.23.65.67@KRBTEST.COM (short username oryx)
I20260810 18:11:58.245126 4197 webserver.cc:460] Webserver started at http://127.23.65.67:41661/ using document root <none> and password file <none>
I20260810 18:11:58.245639 4197 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:58.245697 4197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:58.245867 4197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:58.247998 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "1b626ebaba5c4745bf7321813aa45ef8"
format_stamp: "Formatted at 2026-08-10 18:11:58 on dist-test-slave-jpj1"
server_key: "ffbc796e1e8cffe708d42a31b1738904"
server_key_iv: "eff61cbb212210d39d3cdcd450d59b89"
server_key_version: "encryptionkey@0"
I20260810 18:11:58.248567 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "1b626ebaba5c4745bf7321813aa45ef8"
format_stamp: "Formatted at 2026-08-10 18:11:58 on dist-test-slave-jpj1"
server_key: "ffbc796e1e8cffe708d42a31b1738904"
server_key_iv: "eff61cbb212210d39d3cdcd450d59b89"
server_key_version: "encryptionkey@0"
I20260810 18:11:58.252782 4197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:11:58.255291 4218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:58.256577 4197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:58.256707 4197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "1b626ebaba5c4745bf7321813aa45ef8"
format_stamp: "Formatted at 2026-08-10 18:11:58 on dist-test-slave-jpj1"
server_key: "ffbc796e1e8cffe708d42a31b1738904"
server_key_iv: "eff61cbb212210d39d3cdcd450d59b89"
server_key_version: "encryptionkey@0"
I20260810 18:11:58.256806 4197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:58.277246 4197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:58.280313 4197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:58.280464 4197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:58.281895 4197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:11:58.281949 4197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:58.281991 4197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:11:58.282021 4197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:11:58 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385518, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:58.293778 4197 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:36073
I20260810 18:11:58.293810 4397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:36073 every 8 connection(s)
I20260810 18:11:58.294852 4197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:11:58.296471 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.283000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40549 (local address 127.23.65.67:36357)
0810 18:11:58.283464 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.283514 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.284388 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.285171 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.285180 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.285510 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:58.286031 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.286048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.286928 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.286932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.287935 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.287947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.288514 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.288519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.288676 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.289251 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:58.289267 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:58.291666 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:58.293684 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.293690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.293705 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.294014 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.294329 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.294331 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.294332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.294461 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.294892 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:58.294897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:58.295152 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.295336 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.295698 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260810 18:11:58.297066 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.282541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:36357 (local address 127.23.65.126:40549)
0810 18:11:58.282679 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.282684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.283644 (+ 960us) server_negotiation.cc:407] Connection header received
0810 18:11:58.284638 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.284642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.284700 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.284760 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:58.286179 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.286802 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.288056 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.288338 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.288431 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.291866 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:58.291871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:58.291874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:58.291906 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:58.293573 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.294139 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.294142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.294144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.294245 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.294562 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.294565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.294567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.294803 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:58.294912 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.296846 (+ 1934us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.296954 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:11:58.297991 4399 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40549
I20260810 18:11:58.298141 4399 heartbeater.cc:464] Registering TS with master...
I20260810 18:11:58.298601 4399 heartbeater.cc:511] Master 127.23.65.126:40549 requested a full tablet report, sending...
I20260810 18:11:58.299641 3621 ts_manager.cc:194] Registered new tserver with Master: 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073)
I20260810 18:11:58.300192 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 4197
I20260810 18:11:58.300324 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:11:58.300320 3621 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:36357
I20260810 18:11:58.300668 23813 external_mini_cluster.cc:1428] Setting key d596534434a6d5cd22fe001b9b59a32e
I20260810 18:11:58.302603 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:11:58.307332 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.300331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:45633 (local address 127.23.65.126:40549)
0810 18:11:58.300447 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.300451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.300463 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:11:58.300498 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.300500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.300542 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.300614 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:58.301249 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.301973 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.302935 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.303258 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.303367 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.304446 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:58.304450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:58.304452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:58.304476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:58.305894 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.306271 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.306274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.306276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.306352 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.306595 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.306598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.306600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.306761 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:58.306853 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.307098 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.307198 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:11:58.307344 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.299999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40549 (local address 127.23.65.67:45633)
0810 18:11:58.300287 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.300302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.300389 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.300775 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.300778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.300970 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:58.301095 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.301102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.302128 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.302132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.302830 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.302840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.303398 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.303401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.303483 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.303980 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:58.303991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:58.304320 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:58.305992 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.305995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.305997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.306184 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.306432 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.306435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.306437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.306517 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.306856 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:58.306860 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:58.306946 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.307032 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.307164 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
Aug 10 18:11:58 dist-test-slave-jpj1 krb5kdc[3559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385518, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:11:58 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:11:58 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:11:58.327843 3701 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:58.321866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.23.65.126:40549)
0810 18:11:58.322013 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.322016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.322028 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:11:58.322065 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.322067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.322105 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.322172 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:58.322880 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.323395 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.324316 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.324534 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.324624 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.327668 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47024: BlockingRecv error: recv got EOF from 127.0.0.1:47024 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 18:11:58 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385518, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:58.336942 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.329150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.23.65.126:40549)
0810 18:11:58.329259 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.329262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.329389 (+ 127us) server_negotiation.cc:407] Connection header received
0810 18:11:58.329497 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.329499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.329540 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.329628 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:58.330171 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.330673 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.331472 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.331741 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.331824 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.334162 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:58.334165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:58.334167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:58.334198 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:58.335654 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.336025 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.336027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.336028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.336098 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.336277 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.336279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.336280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.336444 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:58.336541 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.336708 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.336817 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:11:58.339767 3621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:58.342886 3621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:58.356472 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.351137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:36073 (local address 127.0.0.1:35374)
0810 18:11:58.351474 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.351518 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.351658 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.352434 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.352439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.352480 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:58.352614 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.352621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.353768 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.353771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.354656 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.354666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.355944 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.355948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.356057 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.356095 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.356201 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.356320 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
I20260810 18:11:58.356513 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.351301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.23.65.67:36073)
0810 18:11:58.351661 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.351672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.351694 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:11:58.351866 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.351871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.352088 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.352251 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:58.352752 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.353676 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.354777 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.355611 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.355711 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.355881 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.356258 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.356367 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260810 18:11:58.357487 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.352228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:46099 (local address 127.0.0.1:33728)
0810 18:11:58.352443 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.352456 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.352543 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.353560 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.353563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.353580 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:58.353681 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.353689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.355122 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.355125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.356111 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.356120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.357081 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.357085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.357179 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.357202 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.357281 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.357352 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260810 18:11:58.357507 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.352781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.23.65.65:46099)
0810 18:11:58.353154 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.353167 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.353186 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:11:58.353252 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.353256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.353417 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.353842 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:58.353972 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.354998 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.356253 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.356967 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.357065 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.357220 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.357301 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.357385 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":88,"threads_started":1}
I20260810 18:11:58.358094 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.351883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:35857 (local address 127.0.0.1:55426)
0810 18:11:58.352201 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.352213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.352289 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.353429 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.353432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.353452 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:58.353559 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.353565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.354921 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.354924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.356225 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.356234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.357721 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.357725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.357814 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.357838 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.357907 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.357971 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":170,"thread_start_us":113,"threads_started":1}
I20260810 18:11:58.358515 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.352055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.23.65.66:35857)
0810 18:11:58.352968 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.352980 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.353003 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:11:58.353077 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.353082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.353282 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.353433 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:58.353675 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.354789 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.356360 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.357393 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.357499 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.357656 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.358265 (+ 609us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.358373 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":121,"threads_started":1}
I20260810 18:11:58.359669 4292 tablet_service.cc:1467] Processing CreateTablet for tablet a45692bbb5f245df8074144661db0dc9 (DEFAULT_TABLE table=test-table [id=513825cfbc1748b08d2780c427ef3281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:58.360996 4292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45692bbb5f245df8074144661db0dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:58.362075 3841 tablet_service.cc:1467] Processing CreateTablet for tablet a45692bbb5f245df8074144661db0dc9 (DEFAULT_TABLE table=test-table [id=513825cfbc1748b08d2780c427ef3281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:58.362345 4069 tablet_service.cc:1467] Processing CreateTablet for tablet a45692bbb5f245df8074144661db0dc9 (DEFAULT_TABLE table=test-table [id=513825cfbc1748b08d2780c427ef3281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:58.362929 3841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45692bbb5f245df8074144661db0dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:58.363379 4069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45692bbb5f245df8074144661db0dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:58.369836 4440 tablet_bootstrap.cc:492] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: Bootstrap starting.
I20260810 18:11:58.369988 4441 tablet_bootstrap.cc:492] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: Bootstrap starting.
I20260810 18:11:58.371148 4443 tablet_bootstrap.cc:492] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: Bootstrap starting.
I20260810 18:11:58.372375 4441 tablet_bootstrap.cc:654] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:58.372531 4440 tablet_bootstrap.cc:654] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:58.373160 4441 log.cc:824] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:58.373310 4443 tablet_bootstrap.cc:654] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:58.373842 4440 log.cc:824] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:58.374066 4443 log.cc:824] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:58.375174 4441 tablet_bootstrap.cc:492] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: No bootstrap required, opened a new log
I20260810 18:11:58.375370 4441 ts_tablet_manager.cc:1397] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:11:58.375766 4440 tablet_bootstrap.cc:492] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: No bootstrap required, opened a new log
I20260810 18:11:58.375952 4440 ts_tablet_manager.cc:1397] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260810 18:11:58.376375 4443 tablet_bootstrap.cc:492] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: No bootstrap required, opened a new log
I20260810 18:11:58.376559 4443 ts_tablet_manager.cc:1397] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:11:58.378559 4441 raft_consensus.cc:348] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } }
I20260810 18:11:58.378784 4441 raft_consensus.cc:374] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:58.378845 4441 raft_consensus.cc:729] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b626ebaba5c4745bf7321813aa45ef8, State: Initialized, Role: FOLLOWER
I20260810 18:11:58.379256 4441 consensus_queue.cc:260] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } }
I20260810 18:11:58.379949 4399 heartbeater.cc:503] Master 127.23.65.126:40549 was elected leader, sending a full tablet report...
I20260810 18:11:58.380244 4441 ts_tablet_manager.cc:1428] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260810 18:11:58.380240 4443 raft_consensus.cc:348] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } }
I20260810 18:11:58.379741 4440 raft_consensus.cc:348] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } }
I20260810 18:11:58.380489 4443 raft_consensus.cc:374] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:58.380550 4443 raft_consensus.cc:729] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb12e0294d14498b8f62452b19bcdb14, State: Initialized, Role: FOLLOWER
I20260810 18:11:58.380549 4440 raft_consensus.cc:374] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:58.380605 4440 raft_consensus.cc:729] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf27092e6a0f4e0b98615b02c1d01728, State: Initialized, Role: FOLLOWER
I20260810 18:11:58.380942 4443 consensus_queue.cc:260] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } }
I20260810 18:11:58.381018 4440 consensus_queue.cc:260] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } }
I20260810 18:11:58.381839 4443 ts_tablet_manager.cc:1428] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:11:58.381831 3949 heartbeater.cc:503] Master 127.23.65.126:40549 was elected leader, sending a full tablet report...
I20260810 18:11:58.381919 4440 ts_tablet_manager.cc:1428] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260810 18:11:58.382246 4180 heartbeater.cc:503] Master 127.23.65.126:40549 was elected leader, sending a full tablet report...
I20260810 18:11:58.384517 4453 raft_consensus.cc:482] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:58.384668 4453 raft_consensus.cc:504] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } }
I20260810 18:11:58.385812 4453 leader_election.cc:288] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073), fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857)
I20260810 18:11:58.390288 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.386081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:50917 (local address 127.23.65.67:36073)
0810 18:11:58.386213 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.386217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.386356 (+ 139us) server_negotiation.cc:407] Connection header received
0810 18:11:58.386472 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.386475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.386514 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.386580 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:58.386955 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.387754 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.389197 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.389784 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.389920 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.389973 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.390108 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.390174 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:11:58.390311 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.386014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:36073 (local address 127.23.65.65:50917)
0810 18:11:58.386261 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.386279 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.386378 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.386696 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.386699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.386716 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:58.386841 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.386848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.387860 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.387862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.389072 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.389083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.389905 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.389908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.389986 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.390014 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.390085 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.390155 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260810 18:11:58.390429 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.386482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:51703 (local address 127.23.65.66:35857)
0810 18:11:58.386593 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.386597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.386920 (+ 323us) server_negotiation.cc:407] Connection header received
0810 18:11:58.387020 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.387022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.387064 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.387139 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:58.387356 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.388026 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.389357 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.389936 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.390040 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.390082 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.390247 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.390311 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:11:58.390645 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.386392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:35857 (local address 127.23.65.65:51703)
0810 18:11:58.386679 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.386691 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.386946 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.387142 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.387144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.387155 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:58.387248 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.387251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.388147 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.388151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.389246 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.389253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.390033 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.390036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.390142 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.390162 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.390227 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.390295 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260810 18:11:58.390894 4322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45692bbb5f245df8074144661db0dc9" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8" is_pre_election: true
I20260810 18:11:58.391151 4322 raft_consensus.cc:2454] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 0.
I20260810 18:11:58.391129 4102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45692bbb5f245df8074144661db0dc9" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb12e0294d14498b8f62452b19bcdb14" is_pre_election: true
I20260810 18:11:58.391362 4102 raft_consensus.cc:2454] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 0.
I20260810 18:11:58.391678 3770 leader_election.cc:302] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b626ebaba5c4745bf7321813aa45ef8, cf27092e6a0f4e0b98615b02c1d01728; no voters:
I20260810 18:11:58.391906 4453 raft_consensus.cc:2781] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:58.391994 4453 raft_consensus.cc:482] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:58.392045 4453 raft_consensus.cc:3037] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:58.393000 4453 raft_consensus.cc:504] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } }
I20260810 18:11:58.393280 4453 leader_election.cc:288] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 election: Requested vote from peers 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073), fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857)
I20260810 18:11:58.393563 4322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45692bbb5f245df8074144661db0dc9" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8"
I20260810 18:11:58.393627 4102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45692bbb5f245df8074144661db0dc9" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb12e0294d14498b8f62452b19bcdb14"
I20260810 18:11:58.393676 4322 raft_consensus.cc:3037] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:58.393731 4102 raft_consensus.cc:3037] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:58.394666 4322 raft_consensus.cc:2454] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 1.
I20260810 18:11:58.394961 3770 leader_election.cc:302] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b626ebaba5c4745bf7321813aa45ef8, cf27092e6a0f4e0b98615b02c1d01728; no voters:
I20260810 18:11:58.395061 4102 raft_consensus.cc:2454] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 1.
I20260810 18:11:58.395133 4453 raft_consensus.cc:2781] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:58.395332 4453 raft_consensus.cc:686] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 LEADER]: Becoming Leader. State: Replica: cf27092e6a0f4e0b98615b02c1d01728, State: Running, Role: LEADER
I20260810 18:11:58.395676 4453 consensus_queue.cc:237] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } }
I20260810 18:11:58.398538 3621 catalog_manager.cc:5495] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65). New cstate: current_term: 1 leader_uuid: "cf27092e6a0f4e0b98615b02c1d01728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260810 18:11:58.408825 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.405363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.23.65.65:46099)
0810 18:11:58.405489 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.405493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.405525 (+ 32us) server_negotiation.cc:407] Connection header received
0810 18:11:58.405612 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.405615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.405668 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.405749 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:58.405994 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.406492 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.407327 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.407615 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.407706 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.407995 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:58.408435 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:58.408541 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.408614 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.408675 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:11:58.414745 4322 raft_consensus.cc:1261] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Refusing update from remote peer cf27092e6a0f4e0b98615b02c1d01728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:58.414745 4102 raft_consensus.cc:1261] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Refusing update from remote peer cf27092e6a0f4e0b98615b02c1d01728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:58.415395 4453 consensus_queue.cc:1035] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:58.415648 4458 consensus_queue.cc:1035] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:58.425323 4467 mvcc.cc:205] Tried to move back new op lower bound from 7317035083419979776 to 7317035083351113728. Current Snapshot: MvccSnapshot[applied={T|T < 7317035083419979776 or (T in {7317035083419979776})}]
I20260810 18:11:58.430306 3621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:58.430490 3621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:58.432891 3621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: Sending DeleteTablet for 3 replicas of tablet a45692bbb5f245df8074144661db0dc9
I20260810 18:11:58.433624 4292 tablet_service.cc:1514] Processing DeleteTablet for tablet a45692bbb5f245df8074144661db0dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:58 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:35374
I20260810 18:11:58.433691 4069 tablet_service.cc:1514] Processing DeleteTablet for tablet a45692bbb5f245df8074144661db0dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:58 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:55426
I20260810 18:11:58.434166 4475 tablet_replica.cc:330] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: stopping tablet replica
I20260810 18:11:58.434171 3841 tablet_service.cc:1514] Processing DeleteTablet for tablet a45692bbb5f245df8074144661db0dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:58 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:33728
I20260810 18:11:58.434432 4475 raft_consensus.cc:2229] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:58.434645 4476 tablet_replica.cc:330] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: stopping tablet replica
I20260810 18:11:58.434733 4475 raft_consensus.cc:2258] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:58.435092 4474 tablet_replica.cc:330] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: stopping tablet replica
I20260810 18:11:58.435113 4476 raft_consensus.cc:2229] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:58.435386 4474 raft_consensus.cc:2229] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:58.435694 4474 raft_consensus.cc:2258] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:58.436389 4475 ts_tablet_manager.cc:1905] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:58.436561 3621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:58.436991 3621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:58.437135 4474 ts_tablet_manager.cc:1905] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:58.439209 4475 ts_tablet_manager.cc:1918] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:58.439311 4475 log.cc:1196] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/a45692bbb5f245df8074144661db0dc9
I20260810 18:11:58.439654 4475 ts_tablet_manager.cc:1939] T a45692bbb5f245df8074144661db0dc9 P 1b626ebaba5c4745bf7321813aa45ef8: Deleting consensus metadata
I20260810 18:11:58.440260 4474 ts_tablet_manager.cc:1918] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:58.440361 4474 log.cc:1196] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/a45692bbb5f245df8074144661db0dc9
I20260810 18:11:58.440647 4474 ts_tablet_manager.cc:1939] T a45692bbb5f245df8074144661db0dc9 P fb12e0294d14498b8f62452b19bcdb14: Deleting consensus metadata
I20260810 18:11:58.440935 3603 catalog_manager.cc:4841] TS 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073): tablet a45692bbb5f245df8074144661db0dc9 (table test-table [id=513825cfbc1748b08d2780c427ef3281]) successfully deleted
I20260810 18:11:58.441768 3603 catalog_manager.cc:4841] TS fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857): tablet a45692bbb5f245df8074144661db0dc9 (table test-table [id=513825cfbc1748b08d2780c427ef3281]) successfully deleted
I20260810 18:11:58.441804 4476 raft_consensus.cc:2258] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:58.442479 3841 tablet_service.cc:1467] Processing CreateTablet for tablet c7ecdc1136224dc9998d22e16e93baf7 (DEFAULT_TABLE table=test-table [id=f69556c79cff4d8cb4c52c0cd2dfe570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:58.442517 4069 tablet_service.cc:1467] Processing CreateTablet for tablet c7ecdc1136224dc9998d22e16e93baf7 (DEFAULT_TABLE table=test-table [id=f69556c79cff4d8cb4c52c0cd2dfe570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:58.442739 3841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ecdc1136224dc9998d22e16e93baf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:58.442739 4069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ecdc1136224dc9998d22e16e93baf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:58.444907 4443 tablet_bootstrap.cc:492] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: Bootstrap starting.
I20260810 18:11:58.445087 4440 tablet_bootstrap.cc:492] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: Bootstrap starting.
I20260810 18:11:58.445858 4443 tablet_bootstrap.cc:654] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:58.446043 4440 tablet_bootstrap.cc:654] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:58.446759 4292 tablet_service.cc:1467] Processing CreateTablet for tablet c7ecdc1136224dc9998d22e16e93baf7 (DEFAULT_TABLE table=test-table [id=f69556c79cff4d8cb4c52c0cd2dfe570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:58.446874 4476 ts_tablet_manager.cc:1905] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:58.447012 4292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ecdc1136224dc9998d22e16e93baf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:58.448344 4443 tablet_bootstrap.cc:492] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: No bootstrap required, opened a new log
I20260810 18:11:58.448434 4443 ts_tablet_manager.cc:1397] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 18:11:58.448951 4443 raft_consensus.cc:348] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.449077 4443 raft_consensus.cc:374] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:58.449112 4443 raft_consensus.cc:729] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb12e0294d14498b8f62452b19bcdb14, State: Initialized, Role: FOLLOWER
I20260810 18:11:58.449250 4443 consensus_queue.cc:260] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.449280 4476 ts_tablet_manager.cc:1918] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:58.449385 4476 log.cc:1196] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/a45692bbb5f245df8074144661db0dc9
I20260810 18:11:58.449543 4443 ts_tablet_manager.cc:1428] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:58.449744 4476 ts_tablet_manager.cc:1939] T a45692bbb5f245df8074144661db0dc9 P cf27092e6a0f4e0b98615b02c1d01728: Deleting consensus metadata
I20260810 18:11:58.450762 4441 tablet_bootstrap.cc:492] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: Bootstrap starting.
I20260810 18:11:58.451399 3603 catalog_manager.cc:4841] TS cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65:46099): tablet a45692bbb5f245df8074144661db0dc9 (table test-table [id=513825cfbc1748b08d2780c427ef3281]) successfully deleted
I20260810 18:11:58.451836 4441 tablet_bootstrap.cc:654] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:58.452828 4440 tablet_bootstrap.cc:492] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: No bootstrap required, opened a new log
I20260810 18:11:58.452926 4440 ts_tablet_manager.cc:1397] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260810 18:11:58.453426 4440 raft_consensus.cc:348] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.453537 4440 raft_consensus.cc:374] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:58.453559 4440 raft_consensus.cc:729] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf27092e6a0f4e0b98615b02c1d01728, State: Initialized, Role: FOLLOWER
I20260810 18:11:58.453699 4440 consensus_queue.cc:260] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.453928 4440 ts_tablet_manager.cc:1428] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:58.455047 4441 tablet_bootstrap.cc:492] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: No bootstrap required, opened a new log
I20260810 18:11:58.455163 4441 ts_tablet_manager.cc:1397] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260810 18:11:58.455628 4441 raft_consensus.cc:348] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.455747 4441 raft_consensus.cc:374] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:58.455783 4441 raft_consensus.cc:729] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b626ebaba5c4745bf7321813aa45ef8, State: Initialized, Role: FOLLOWER
I20260810 18:11:58.455905 4441 consensus_queue.cc:260] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.456527 4441 ts_tablet_manager.cc:1428] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:58.495199 4452 raft_consensus.cc:482] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:58.495364 4452 raft_consensus.cc:504] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.496747 4452 leader_election.cc:288] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65:46099), 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073)
I20260810 18:11:58.501268 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.497455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:41991 (local address 127.23.65.67:36073)
0810 18:11:58.497592 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.497596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.497717 (+ 121us) server_negotiation.cc:407] Connection header received
0810 18:11:58.497859 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.497862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.497913 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.497986 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:58.498327 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.498945 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.500121 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.500693 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.500853 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.500916 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.501066 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.501139 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:11:58.501279 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.497241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:36073 (local address 127.23.65.66:41991)
0810 18:11:58.497619 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.497633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.497751 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.498007 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.498010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.498027 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:58.498184 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.498191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.499053 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.499057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.499997 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.500008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.500830 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.500834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.500939 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.500967 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.501042 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.501110 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":108,"threads_started":1}
I20260810 18:11:58.501411 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.496968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:46099 (local address 127.23.65.66:36801)
0810 18:11:58.497118 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.497134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.497228 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.497612 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.497615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.497636 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:11:58.497749 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.497756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.498795 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.498798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.500079 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.500091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.501028 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.501031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.501139 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.501162 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.501228 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.501294 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 18:11:58.501685 4322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ecdc1136224dc9998d22e16e93baf7" candidate_uuid: "fb12e0294d14498b8f62452b19bcdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8" is_pre_election: true
I20260810 18:11:58.501786 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.497092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:36801 (local address 127.23.65.65:46099)
0810 18:11:58.497215 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.497220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.497233 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:58.497370 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.497373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.497424 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.497487 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:11:58.497878 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.498679 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.500238 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.500820 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.500922 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.500969 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.501590 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.501672 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:11:58.501850 4322 raft_consensus.cc:2454] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb12e0294d14498b8f62452b19bcdb14 in term 0.
I20260810 18:11:58.502245 4002 leader_election.cc:302] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b626ebaba5c4745bf7321813aa45ef8, fb12e0294d14498b8f62452b19bcdb14; no voters:
I20260810 18:11:58.502414 3872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ecdc1136224dc9998d22e16e93baf7" candidate_uuid: "fb12e0294d14498b8f62452b19bcdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf27092e6a0f4e0b98615b02c1d01728" is_pre_election: true
I20260810 18:11:58.502516 4452 raft_consensus.cc:2781] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:58.502565 4452 raft_consensus.cc:482] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:58.502606 4452 raft_consensus.cc:3037] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:58.502655 3872 raft_consensus.cc:2454] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb12e0294d14498b8f62452b19bcdb14 in term 0.
I20260810 18:11:58.503903 4452 raft_consensus.cc:504] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.504230 4452 leader_election.cc:288] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [CANDIDATE]: Term 1 election: Requested vote from peers cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65:46099), 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073)
I20260810 18:11:58.504619 3872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ecdc1136224dc9998d22e16e93baf7" candidate_uuid: "fb12e0294d14498b8f62452b19bcdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf27092e6a0f4e0b98615b02c1d01728"
I20260810 18:11:58.504619 4322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ecdc1136224dc9998d22e16e93baf7" candidate_uuid: "fb12e0294d14498b8f62452b19bcdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8"
I20260810 18:11:58.504755 4322 raft_consensus.cc:3037] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:58.504755 3872 raft_consensus.cc:3037] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:58.505897 4322 raft_consensus.cc:2454] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb12e0294d14498b8f62452b19bcdb14 in term 1.
I20260810 18:11:58.505897 3872 raft_consensus.cc:2454] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb12e0294d14498b8f62452b19bcdb14 in term 1.
I20260810 18:11:58.506191 4002 leader_election.cc:302] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf27092e6a0f4e0b98615b02c1d01728, fb12e0294d14498b8f62452b19bcdb14; no voters:
I20260810 18:11:58.506366 4452 raft_consensus.cc:2781] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:58.506614 4452 raft_consensus.cc:686] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 1 LEADER]: Becoming Leader. State: Replica: fb12e0294d14498b8f62452b19bcdb14, State: Running, Role: LEADER
I20260810 18:11:58.506878 4452 consensus_queue.cc:237] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.509294 3621 catalog_manager.cc:5495] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "fb12e0294d14498b8f62452b19bcdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } health_report { overall_health: UNKNOWN } } }
W20260810 18:11:58.545887 4400 tablet.cc:2369] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 18:11:58 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385517, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.126@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
W20260810 18:11:58.566452 4182 tablet.cc:2369] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:58.570060 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.561455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40549 (local address 127.0.0.1:47034)
0810 18:11:58.561755 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.561770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.561911 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.562136 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.562138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.562548 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:58.562648 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.562656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.563414 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.563417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.564033 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.564043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.564464 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.564467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.564564 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.565150 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:58.565163 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:58.566598 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:58.568447 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.568457 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.568462 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.568709 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.568988 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:58.568991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:58.568992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:58.569068 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:58.569516 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:58.569522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:58.569656 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.569752 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.569906 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":3,"thread_start_us":124,"threads_started":1}
I20260810 18:11:58.570334 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.561490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.23.65.126:40549)
0810 18:11:58.561633 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.561637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.561855 (+ 218us) server_negotiation.cc:407] Connection header received
0810 18:11:58.561997 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.562000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.562041 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.562104 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:58.562774 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.563302 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.564143 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.564364 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.564449 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.566769 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:58.566774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:58.566777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:58.566806 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:58.568333 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.568804 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.568808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.568809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.568904 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:58.569153 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:58.569156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:58.569158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:58.569397 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:58.569491 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.569821 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.569981 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":41}
I20260810 18:11:58.574316 3621 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:47034:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:11:58.575069 3621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:58.579945 3841 tablet_service.cc:1467] Processing CreateTablet for tablet 47fee75da6fe410ba33862fb25a0c218 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94d542a5bd5e4f6b8d5ddc2e9cea2fbe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:11:58.580201 4292 tablet_service.cc:1467] Processing CreateTablet for tablet 47fee75da6fe410ba33862fb25a0c218 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94d542a5bd5e4f6b8d5ddc2e9cea2fbe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:11:58.580382 3841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47fee75da6fe410ba33862fb25a0c218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:58.580559 4292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47fee75da6fe410ba33862fb25a0c218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:58.583505 4441 tablet_bootstrap.cc:492] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: Bootstrap starting.
I20260810 18:11:58.583384 4069 tablet_service.cc:1467] Processing CreateTablet for tablet 47fee75da6fe410ba33862fb25a0c218 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94d542a5bd5e4f6b8d5ddc2e9cea2fbe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:11:58.583767 4069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47fee75da6fe410ba33862fb25a0c218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:58.584529 4441 tablet_bootstrap.cc:654] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:58.585987 4441 tablet_bootstrap.cc:492] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: No bootstrap required, opened a new log
I20260810 18:11:58.586056 4441 ts_tablet_manager.cc:1397] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:11:58.586449 4441 raft_consensus.cc:348] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.586555 4441 raft_consensus.cc:374] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:58.586580 4441 raft_consensus.cc:729] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b626ebaba5c4745bf7321813aa45ef8, State: Initialized, Role: FOLLOWER
I20260810 18:11:58.586683 4441 consensus_queue.cc:260] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.586886 4443 tablet_bootstrap.cc:492] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: Bootstrap starting.
I20260810 18:11:58.586905 4441 ts_tablet_manager.cc:1428] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:58.587080 4449 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.587226 4449 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:58.587965 4443 tablet_bootstrap.cc:654] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:58.589279 4443 tablet_bootstrap.cc:492] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: No bootstrap required, opened a new log
I20260810 18:11:58.589376 4443 ts_tablet_manager.cc:1397] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:11:58.589790 4443 raft_consensus.cc:348] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.589897 4443 raft_consensus.cc:374] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:58.589932 4443 raft_consensus.cc:729] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb12e0294d14498b8f62452b19bcdb14, State: Initialized, Role: FOLLOWER
I20260810 18:11:58.590041 4443 consensus_queue.cc:260] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.590268 4443 ts_tablet_manager.cc:1428] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:58.590440 4452 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.590560 4452 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:58.590782 4440 tablet_bootstrap.cc:492] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: Bootstrap starting.
I20260810 18:11:58.591672 4440 tablet_bootstrap.cc:654] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:58.592746 4440 tablet_bootstrap.cc:492] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: No bootstrap required, opened a new log
I20260810 18:11:58.592832 4440 ts_tablet_manager.cc:1397] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 18:11:58.593189 4440 raft_consensus.cc:348] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.593297 4440 raft_consensus.cc:374] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:58.593340 4440 raft_consensus.cc:729] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf27092e6a0f4e0b98615b02c1d01728, State: Initialized, Role: FOLLOWER
I20260810 18:11:58.593459 4440 consensus_queue.cc:260] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.593684 4458 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.593827 4458 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:58.593698 4440 ts_tablet_manager.cc:1428] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260810 18:11:58.605569 3952 tablet.cc:2369] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:58.887087 4452 raft_consensus.cc:482] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:58.887226 4452 raft_consensus.cc:504] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.887632 4452 leader_election.cc:288] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65:46099), 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073)
I20260810 18:11:58.888104 3872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fee75da6fe410ba33862fb25a0c218" candidate_uuid: "fb12e0294d14498b8f62452b19bcdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf27092e6a0f4e0b98615b02c1d01728" is_pre_election: true
I20260810 18:11:58.888170 4322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fee75da6fe410ba33862fb25a0c218" candidate_uuid: "fb12e0294d14498b8f62452b19bcdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8" is_pre_election: true
I20260810 18:11:58.888257 3872 raft_consensus.cc:2454] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb12e0294d14498b8f62452b19bcdb14 in term 0.
I20260810 18:11:58.888350 4322 raft_consensus.cc:2454] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb12e0294d14498b8f62452b19bcdb14 in term 0.
I20260810 18:11:58.888574 4002 leader_election.cc:302] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf27092e6a0f4e0b98615b02c1d01728, fb12e0294d14498b8f62452b19bcdb14; no voters:
I20260810 18:11:58.888756 4452 raft_consensus.cc:2781] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:58.888834 4452 raft_consensus.cc:482] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:58.888878 4452 raft_consensus.cc:3037] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:58.889914 4452 raft_consensus.cc:504] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.890254 4452 leader_election.cc:288] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [CANDIDATE]: Term 1 election: Requested vote from peers cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65:46099), 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073)
I20260810 18:11:58.890679 4322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fee75da6fe410ba33862fb25a0c218" candidate_uuid: "fb12e0294d14498b8f62452b19bcdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8"
I20260810 18:11:58.890712 3872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fee75da6fe410ba33862fb25a0c218" candidate_uuid: "fb12e0294d14498b8f62452b19bcdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf27092e6a0f4e0b98615b02c1d01728"
I20260810 18:11:58.890810 4322 raft_consensus.cc:3037] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:58.890861 3872 raft_consensus.cc:3037] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:58.891798 4322 raft_consensus.cc:2454] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb12e0294d14498b8f62452b19bcdb14 in term 1.
I20260810 18:11:58.891920 3872 raft_consensus.cc:2454] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb12e0294d14498b8f62452b19bcdb14 in term 1.
I20260810 18:11:58.892114 4002 leader_election.cc:302] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b626ebaba5c4745bf7321813aa45ef8, fb12e0294d14498b8f62452b19bcdb14; no voters:
I20260810 18:11:58.892282 4452 raft_consensus.cc:2781] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:58.892405 4452 raft_consensus.cc:686] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [term 1 LEADER]: Becoming Leader. State: Replica: fb12e0294d14498b8f62452b19bcdb14, State: Running, Role: LEADER
I20260810 18:11:58.892544 4452 consensus_queue.cc:237] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:58.893179 4486 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: TxnStatusTablet state changed. Reason: New leader fb12e0294d14498b8f62452b19bcdb14. Latest consensus state: current_term: 1 leader_uuid: "fb12e0294d14498b8f62452b19bcdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.893285 4486 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:11:58.893512 4507 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:11:58.893654 4507 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:11:58.894201 3620 catalog_manager.cc:5495] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "fb12e0294d14498b8f62452b19bcdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } health_report { overall_health: UNKNOWN } } }
I20260810 18:11:58.960397 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.956484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:35857 (local address 127.0.0.1:55436)
0810 18:11:58.956631 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.956648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.956747 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.957128 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.957131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.957139 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:58.957239 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.957245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.958007 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.958011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.958634 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.958643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.959198 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.959201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.959306 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.959357 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:58.960103 (+ 746us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:58.960121 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.960186 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.960269 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 18:11:58.960783 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.956619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.23.65.66:35857)
0810 18:11:58.956873 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.956878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.956890 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:11:58.956955 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.956957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.957011 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.957102 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:58.957345 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.957901 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.958751 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.959075 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.959187 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.959507 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:58.960012 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:58.960128 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.960580 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.960640 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
I20260810 18:11:58.963660 4322 raft_consensus.cc:1261] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Refusing update from remote peer fb12e0294d14498b8f62452b19bcdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:58.963871 3872 raft_consensus.cc:1261] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Refusing update from remote peer fb12e0294d14498b8f62452b19bcdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:58.964166 4452 consensus_queue.cc:1035] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:58.964438 4486 consensus_queue.cc:1035] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:58.966101 4458 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: TxnStatusTablet state changed. Reason: New leader fb12e0294d14498b8f62452b19bcdb14. Latest consensus state: current_term: 1 leader_uuid: "fb12e0294d14498b8f62452b19bcdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.966225 4458 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:58.967851 4449 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: TxnStatusTablet state changed. Reason: New leader fb12e0294d14498b8f62452b19bcdb14. Latest consensus state: current_term: 1 leader_uuid: "fb12e0294d14498b8f62452b19bcdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.967976 4449 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:58.968674 4486 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb12e0294d14498b8f62452b19bcdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.968788 4486 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:11:58.968916 4486 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb12e0294d14498b8f62452b19bcdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.968993 4486 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P fb12e0294d14498b8f62452b19bcdb14: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:11:58.969087 4513 mvcc.cc:205] Tried to move back new op lower bound from 7317035085669904384 to 7317035085385117696. Current Snapshot: MvccSnapshot[applied={T|T < 7317035085669904384}]
I20260810 18:11:58.969565 4458 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb12e0294d14498b8f62452b19bcdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.969694 4458 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P cf27092e6a0f4e0b98615b02c1d01728: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:58.970342 4516 mvcc.cc:205] Tried to move back new op lower bound from 7317035085669904384 to 7317035085385117696. Current Snapshot: MvccSnapshot[applied={T|T < 7317035085669904384 or (T in {7317035085669904384})}]
I20260810 18:11:58.971743 4449 tablet_replica.cc:439] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb12e0294d14498b8f62452b19bcdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } } }
I20260810 18:11:58.971864 4449 tablet_replica.cc:442] T 47fee75da6fe410ba33862fb25a0c218 P 1b626ebaba5c4745bf7321813aa45ef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:11:58.978343 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.974826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.23.65.66:35857)
0810 18:11:58.974941 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.974945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.975160 (+ 215us) server_negotiation.cc:407] Connection header received
0810 18:11:58.975278 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.975280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.975329 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.975387 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:58.975687 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.976200 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.977402 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.977604 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.977671 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.977948 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:58.978018 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:58.978062 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.978171 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.978212 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:11:58.988593 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.985541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:58921 (local address 127.23.65.66:35857)
0810 18:11:58.985641 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:58.985646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:58.985834 (+ 188us) server_negotiation.cc:407] Connection header received
0810 18:11:58.985972 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:58.985975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:58.986014 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:58.986068 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:58.986277 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.986755 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.987610 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.987817 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.987903 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.988163 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:58.988229 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:58.988283 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:58.988371 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 18:11:58.988439 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 18:11:58.988623 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:58.985480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:35857 (local address 127.23.65.66:58921)
0810 18:11:58.985760 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:58.985772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:58.985879 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:58.986089 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:58.986092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:58.986099 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:58.986186 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.986191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.986865 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.986868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.987482 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:58.987491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:58.987924 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:58.987927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:58.988008 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:58.988047 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:58.988297 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:58.988303 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:11:58.988362 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 18:11:58.988446 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260810 18:11:58.995940 4322 raft_consensus.cc:1261] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Refusing update from remote peer fb12e0294d14498b8f62452b19bcdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:58.996135 3872 raft_consensus.cc:1261] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Refusing update from remote peer fb12e0294d14498b8f62452b19bcdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:58.996505 4452 consensus_queue.cc:1035] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:58.996650 4486 consensus_queue.cc:1035] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:59.038930 3621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:59.039100 3621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:59.041203 3621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: Sending DeleteTablet for 3 replicas of tablet c7ecdc1136224dc9998d22e16e93baf7
I20260810 18:11:59.041742 4068 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ecdc1136224dc9998d22e16e93baf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:59 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:55426
I20260810 18:11:59.041742 3841 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ecdc1136224dc9998d22e16e93baf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:59 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:33728
I20260810 18:11:59.041863 4292 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ecdc1136224dc9998d22e16e93baf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:59 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:35374
I20260810 18:11:59.042132 4539 tablet_replica.cc:330] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: stopping tablet replica
I20260810 18:11:59.042161 4538 tablet_replica.cc:330] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: stopping tablet replica
I20260810 18:11:59.042268 4538 raft_consensus.cc:2229] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:59.042321 4539 raft_consensus.cc:2229] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:59.042392 4538 raft_consensus.cc:2258] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:59.042711 4542 tablet_replica.cc:330] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: stopping tablet replica
I20260810 18:11:59.042865 4542 raft_consensus.cc:2229] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:59.042919 4539 raft_consensus.cc:2258] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:59.043695 4539 ts_tablet_manager.cc:1905] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:59.045751 4542 raft_consensus.cc:2258] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:59.046532 4542 ts_tablet_manager.cc:1905] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:59.046736 4539 ts_tablet_manager.cc:1918] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:59.046825 4539 log.cc:1196] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/c7ecdc1136224dc9998d22e16e93baf7
I20260810 18:11:59.047077 4539 ts_tablet_manager.cc:1939] T c7ecdc1136224dc9998d22e16e93baf7 P fb12e0294d14498b8f62452b19bcdb14: Deleting consensus metadata
I20260810 18:11:59.047397 4538 ts_tablet_manager.cc:1905] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:59.047829 3603 catalog_manager.cc:4841] TS fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857): tablet c7ecdc1136224dc9998d22e16e93baf7 (table test-table [id=f69556c79cff4d8cb4c52c0cd2dfe570]) successfully deleted
I20260810 18:11:59.049850 4538 ts_tablet_manager.cc:1918] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:59.049860 4542 ts_tablet_manager.cc:1918] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:59.049929 4542 log.cc:1196] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/c7ecdc1136224dc9998d22e16e93baf7
I20260810 18:11:59.049937 4538 log.cc:1196] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/c7ecdc1136224dc9998d22e16e93baf7
I20260810 18:11:59.050163 4542 ts_tablet_manager.cc:1939] T c7ecdc1136224dc9998d22e16e93baf7 P 1b626ebaba5c4745bf7321813aa45ef8: Deleting consensus metadata
I20260810 18:11:59.050194 4538 ts_tablet_manager.cc:1939] T c7ecdc1136224dc9998d22e16e93baf7 P cf27092e6a0f4e0b98615b02c1d01728: Deleting consensus metadata
I20260810 18:11:59.050974 3603 catalog_manager.cc:4841] TS cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65:46099): tablet c7ecdc1136224dc9998d22e16e93baf7 (table test-table [id=f69556c79cff4d8cb4c52c0cd2dfe570]) successfully deleted
I20260810 18:11:59.051164 3603 catalog_manager.cc:4841] TS 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073): tablet c7ecdc1136224dc9998d22e16e93baf7 (table test-table [id=f69556c79cff4d8cb4c52c0cd2dfe570]) successfully deleted
Aug 10 18:11:59 dist-test-slave-jpj1 krb5kdc[3559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:11:59 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:11:59 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:11:59.083877 4553 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:59.073001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.23.65.126:40549)
0810 18:11:59.073228 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:59.073232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:59.075884 (+ 2652us) server_negotiation.cc:407] Connection header received
0810 18:11:59.076017 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:59.076020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:59.076082 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:59.076315 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:59.077152 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.078007 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.079215 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.079650 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.079753 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:59.083724 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47046: BlockingRecv error: recv got EOF from 127.0.0.1:47046 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
Aug 10 18:11:59 dist-test-slave-jpj1 krb5kdc[3559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM
I20260810 18:11:59.094949 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:59.085315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.23.65.126:40549)
0810 18:11:59.085424 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:59.085428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:59.085572 (+ 144us) server_negotiation.cc:407] Connection header received
0810 18:11:59.085680 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:59.085682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:59.085729 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:59.085810 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:59.086444 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.087227 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.088431 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.088792 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.088916 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:59.091447 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:59.091452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:59.091455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:59.091490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:59.093153 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:59.093565 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:59.093568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:59.093570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:59.093665 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:59.093928 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:59.093931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:59.093933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:59.094165 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:59.094311 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:59.094667 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 18:11:59.094786 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:11:59.097621 3621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:59.098078 3621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:59.103103 3841 tablet_service.cc:1467] Processing CreateTablet for tablet e65424fad30a45d3be2b3a505a9ff897 (DEFAULT_TABLE table=test-table [id=423f35d35897499b9d399437514d76ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:59.103135 4068 tablet_service.cc:1467] Processing CreateTablet for tablet e65424fad30a45d3be2b3a505a9ff897 (DEFAULT_TABLE table=test-table [id=423f35d35897499b9d399437514d76ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:59.103446 3841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65424fad30a45d3be2b3a505a9ff897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:59.103482 4068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65424fad30a45d3be2b3a505a9ff897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:59.103854 4292 tablet_service.cc:1467] Processing CreateTablet for tablet e65424fad30a45d3be2b3a505a9ff897 (DEFAULT_TABLE table=test-table [id=423f35d35897499b9d399437514d76ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:59.104131 4292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65424fad30a45d3be2b3a505a9ff897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:59.106088 4561 tablet_bootstrap.cc:492] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: Bootstrap starting.
I20260810 18:11:59.107002 4561 tablet_bootstrap.cc:654] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:59.107002 4562 tablet_bootstrap.cc:492] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: Bootstrap starting.
I20260810 18:11:59.107892 4562 tablet_bootstrap.cc:654] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:59.108902 4562 tablet_bootstrap.cc:492] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: No bootstrap required, opened a new log
I20260810 18:11:59.108986 4562 ts_tablet_manager.cc:1397] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:11:59.109198 4565 tablet_bootstrap.cc:492] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: Bootstrap starting.
I20260810 18:11:59.109709 4562 raft_consensus.cc:348] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.109843 4562 raft_consensus.cc:374] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:59.109880 4562 raft_consensus.cc:729] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b626ebaba5c4745bf7321813aa45ef8, State: Initialized, Role: FOLLOWER
I20260810 18:11:59.110030 4562 consensus_queue.cc:260] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.110414 4562 ts_tablet_manager.cc:1428] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:59.110641 4561 tablet_bootstrap.cc:492] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: No bootstrap required, opened a new log
I20260810 18:11:59.110704 4561 ts_tablet_manager.cc:1397] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 18:11:59.111001 4561 raft_consensus.cc:348] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.111076 4561 raft_consensus.cc:374] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:59.111095 4561 raft_consensus.cc:729] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb12e0294d14498b8f62452b19bcdb14, State: Initialized, Role: FOLLOWER
I20260810 18:11:59.111197 4561 consensus_queue.cc:260] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.111449 4561 ts_tablet_manager.cc:1428] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:11:59.111887 4565 tablet_bootstrap.cc:654] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:59.114203 4565 tablet_bootstrap.cc:492] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: No bootstrap required, opened a new log
I20260810 18:11:59.114363 4565 ts_tablet_manager.cc:1397] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 18:11:59.114825 4565 raft_consensus.cc:348] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.114948 4565 raft_consensus.cc:374] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:59.114989 4565 raft_consensus.cc:729] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf27092e6a0f4e0b98615b02c1d01728, State: Initialized, Role: FOLLOWER
I20260810 18:11:59.115114 4565 consensus_queue.cc:260] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.115475 4565 ts_tablet_manager.cc:1428] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:59.189641 4458 raft_consensus.cc:482] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:59.189785 4458 raft_consensus.cc:504] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.190229 4458 leader_election.cc:288] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857), 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073)
I20260810 18:11:59.190712 4102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65424fad30a45d3be2b3a505a9ff897" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb12e0294d14498b8f62452b19bcdb14" is_pre_election: true
I20260810 18:11:59.190847 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65424fad30a45d3be2b3a505a9ff897" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8" is_pre_election: true
I20260810 18:11:59.190892 4102 raft_consensus.cc:2454] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 0.
I20260810 18:11:59.190992 4317 raft_consensus.cc:2454] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 0.
I20260810 18:11:59.191228 3770 leader_election.cc:302] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf27092e6a0f4e0b98615b02c1d01728, fb12e0294d14498b8f62452b19bcdb14; no voters:
I20260810 18:11:59.191460 4458 raft_consensus.cc:2781] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:59.191526 4458 raft_consensus.cc:482] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:59.191598 4458 raft_consensus.cc:3037] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:59.193017 4458 raft_consensus.cc:504] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.193440 4458 leader_election.cc:288] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 election: Requested vote from peers fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857), 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073)
I20260810 18:11:59.193845 4102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65424fad30a45d3be2b3a505a9ff897" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb12e0294d14498b8f62452b19bcdb14"
I20260810 18:11:59.193837 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65424fad30a45d3be2b3a505a9ff897" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8"
I20260810 18:11:59.193966 4317 raft_consensus.cc:3037] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:59.193966 4102 raft_consensus.cc:3037] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:59.195164 4317 raft_consensus.cc:2454] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 1.
I20260810 18:11:59.195164 4102 raft_consensus.cc:2454] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 1.
I20260810 18:11:59.195485 3770 leader_election.cc:302] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b626ebaba5c4745bf7321813aa45ef8, cf27092e6a0f4e0b98615b02c1d01728; no voters:
I20260810 18:11:59.195706 4458 raft_consensus.cc:2781] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:59.195878 4458 raft_consensus.cc:686] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 LEADER]: Becoming Leader. State: Replica: cf27092e6a0f4e0b98615b02c1d01728, State: Running, Role: LEADER
I20260810 18:11:59.196251 4458 consensus_queue.cc:237] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.198112 3620 catalog_manager.cc:5495] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65). New cstate: current_term: 1 leader_uuid: "cf27092e6a0f4e0b98615b02c1d01728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } health_report { overall_health: UNKNOWN } } }
I20260810 18:11:59.223668 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:59.220309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.23.65.65:46099)
0810 18:11:59.220638 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:59.220644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:59.220657 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:59.220704 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:59.220706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:59.220750 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:59.220921 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:59.221178 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.221727 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.222603 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.222843 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.222931 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:59.223175 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:59.223255 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:59.223315 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:59.223421 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 18:11:59.223481 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
I20260810 18:11:59.225651 4317 raft_consensus.cc:1261] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Refusing update from remote peer cf27092e6a0f4e0b98615b02c1d01728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:59.225704 4102 raft_consensus.cc:1261] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Refusing update from remote peer cf27092e6a0f4e0b98615b02c1d01728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:59.226071 4458 consensus_queue.cc:1035] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:59.226234 4569 consensus_queue.cc:1035] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:59.232904 3620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:59.233732 3620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:59.235709 3620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: Sending DeleteTablet for 3 replicas of tablet e65424fad30a45d3be2b3a505a9ff897
I20260810 18:11:59.236227 3841 tablet_service.cc:1514] Processing DeleteTablet for tablet e65424fad30a45d3be2b3a505a9ff897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:59 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:33728
I20260810 18:11:59.236428 4538 tablet_replica.cc:330] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: stopping tablet replica
I20260810 18:11:59.236577 4538 raft_consensus.cc:2229] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:59.236694 4292 tablet_service.cc:1514] Processing DeleteTablet for tablet e65424fad30a45d3be2b3a505a9ff897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:59 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:35374
I20260810 18:11:59.236774 4538 raft_consensus.cc:2258] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:59.236918 4542 tablet_replica.cc:330] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: stopping tablet replica
I20260810 18:11:59.236979 4068 tablet_service.cc:1514] Processing DeleteTablet for tablet e65424fad30a45d3be2b3a505a9ff897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:59 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:55426
I20260810 18:11:59.237035 4542 raft_consensus.cc:2229] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:59.237157 4542 raft_consensus.cc:2258] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:59.237166 4539 tablet_replica.cc:330] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: stopping tablet replica
I20260810 18:11:59.237295 4539 raft_consensus.cc:2229] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:59.237429 4539 raft_consensus.cc:2258] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:59.237612 4538 ts_tablet_manager.cc:1905] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:59.237910 4542 ts_tablet_manager.cc:1905] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:59.238073 4539 ts_tablet_manager.cc:1905] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:59.239817 4542 ts_tablet_manager.cc:1918] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:59.239890 4542 log.cc:1196] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/e65424fad30a45d3be2b3a505a9ff897
I20260810 18:11:59.240108 4542 ts_tablet_manager.cc:1939] T e65424fad30a45d3be2b3a505a9ff897 P 1b626ebaba5c4745bf7321813aa45ef8: Deleting consensus metadata
I20260810 18:11:59.240260 4538 ts_tablet_manager.cc:1918] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:59.240350 4538 log.cc:1196] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/e65424fad30a45d3be2b3a505a9ff897
I20260810 18:11:59.240442 4539 ts_tablet_manager.cc:1918] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:11:59.240514 4539 log.cc:1196] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/e65424fad30a45d3be2b3a505a9ff897
I20260810 18:11:59.240612 4538 ts_tablet_manager.cc:1939] T e65424fad30a45d3be2b3a505a9ff897 P cf27092e6a0f4e0b98615b02c1d01728: Deleting consensus metadata
I20260810 18:11:59.240734 4539 ts_tablet_manager.cc:1939] T e65424fad30a45d3be2b3a505a9ff897 P fb12e0294d14498b8f62452b19bcdb14: Deleting consensus metadata
I20260810 18:11:59.240724 3603 catalog_manager.cc:4841] TS 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073): tablet e65424fad30a45d3be2b3a505a9ff897 (table test-table [id=423f35d35897499b9d399437514d76ab]) successfully deleted
I20260810 18:11:59.241313 3603 catalog_manager.cc:4841] TS fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857): tablet e65424fad30a45d3be2b3a505a9ff897 (table test-table [id=423f35d35897499b9d399437514d76ab]) successfully deleted
I20260810 18:11:59.241451 3603 catalog_manager.cc:4841] TS cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65:46099): tablet e65424fad30a45d3be2b3a505a9ff897 (table test-table [id=423f35d35897499b9d399437514d76ab]) successfully deleted
I20260810 18:11:59.242245 3620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:11:59.242607 3620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:11:59.246721 3841 tablet_service.cc:1467] Processing CreateTablet for tablet f71bcff179f5472ebecc3c92011b03bd (DEFAULT_TABLE table=test-table [id=18c9b926bed04b659e1909e63150c833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:59.246788 4068 tablet_service.cc:1467] Processing CreateTablet for tablet f71bcff179f5472ebecc3c92011b03bd (DEFAULT_TABLE table=test-table [id=18c9b926bed04b659e1909e63150c833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:59.246878 4292 tablet_service.cc:1467] Processing CreateTablet for tablet f71bcff179f5472ebecc3c92011b03bd (DEFAULT_TABLE table=test-table [id=18c9b926bed04b659e1909e63150c833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:11:59.246994 3841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71bcff179f5472ebecc3c92011b03bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:59.247064 4068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71bcff179f5472ebecc3c92011b03bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:59.247118 4292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71bcff179f5472ebecc3c92011b03bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:59.248917 4561 tablet_bootstrap.cc:492] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: Bootstrap starting.
I20260810 18:11:59.249353 4562 tablet_bootstrap.cc:492] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: Bootstrap starting.
I20260810 18:11:59.249406 4565 tablet_bootstrap.cc:492] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728: Bootstrap starting.
I20260810 18:11:59.249830 4561 tablet_bootstrap.cc:654] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:59.250238 4565 tablet_bootstrap.cc:654] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:59.250734 4562 tablet_bootstrap.cc:654] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:59.251502 4561 tablet_bootstrap.cc:492] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: No bootstrap required, opened a new log
I20260810 18:11:59.251619 4561 ts_tablet_manager.cc:1397] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:11:59.251807 4565 tablet_bootstrap.cc:492] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728: No bootstrap required, opened a new log
I20260810 18:11:59.251935 4565 ts_tablet_manager.cc:1397] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:11:59.252077 4561 raft_consensus.cc:348] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.252198 4561 raft_consensus.cc:374] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:59.252233 4561 raft_consensus.cc:729] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb12e0294d14498b8f62452b19bcdb14, State: Initialized, Role: FOLLOWER
I20260810 18:11:59.252360 4561 consensus_queue.cc:260] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.252645 4561 ts_tablet_manager.cc:1428] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:59.253065 4562 tablet_bootstrap.cc:492] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: No bootstrap required, opened a new log
I20260810 18:11:59.253163 4562 ts_tablet_manager.cc:1397] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 18:11:59.253614 4562 raft_consensus.cc:348] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.253734 4562 raft_consensus.cc:374] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:59.253772 4562 raft_consensus.cc:729] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b626ebaba5c4745bf7321813aa45ef8, State: Initialized, Role: FOLLOWER
I20260810 18:11:59.253890 4562 consensus_queue.cc:260] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.254148 4562 ts_tablet_manager.cc:1428] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:11:59.255007 4565 raft_consensus.cc:348] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.255149 4565 raft_consensus.cc:374] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:59.255192 4565 raft_consensus.cc:729] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf27092e6a0f4e0b98615b02c1d01728, State: Initialized, Role: FOLLOWER
I20260810 18:11:59.255333 4565 consensus_queue.cc:260] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.255663 4565 ts_tablet_manager.cc:1428] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 18:11:59.401896 4576 raft_consensus.cc:482] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:11:59.402034 4576 raft_consensus.cc:504] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.402436 4576 leader_election.cc:288] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857), 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073)
I20260810 18:11:59.402911 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71bcff179f5472ebecc3c92011b03bd" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8" is_pre_election: true
I20260810 18:11:59.402890 4102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71bcff179f5472ebecc3c92011b03bd" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb12e0294d14498b8f62452b19bcdb14" is_pre_election: true
I20260810 18:11:59.403051 4317 raft_consensus.cc:2454] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 0.
I20260810 18:11:59.403051 4102 raft_consensus.cc:2454] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 0.
I20260810 18:11:59.403353 3770 leader_election.cc:302] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b626ebaba5c4745bf7321813aa45ef8, cf27092e6a0f4e0b98615b02c1d01728; no voters:
I20260810 18:11:59.403568 4576 raft_consensus.cc:2781] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:11:59.403649 4576 raft_consensus.cc:482] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:11:59.403695 4576 raft_consensus.cc:3037] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:59.405052 4576 raft_consensus.cc:504] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.405428 4576 leader_election.cc:288] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 election: Requested vote from peers fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857), 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073)
I20260810 18:11:59.405861 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71bcff179f5472ebecc3c92011b03bd" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b626ebaba5c4745bf7321813aa45ef8"
I20260810 18:11:59.405874 4102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71bcff179f5472ebecc3c92011b03bd" candidate_uuid: "cf27092e6a0f4e0b98615b02c1d01728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb12e0294d14498b8f62452b19bcdb14"
I20260810 18:11:59.405992 4317 raft_consensus.cc:3037] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:59.406008 4102 raft_consensus.cc:3037] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:59.407095 4317 raft_consensus.cc:2454] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 1.
I20260810 18:11:59.407137 4102 raft_consensus.cc:2454] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf27092e6a0f4e0b98615b02c1d01728 in term 1.
I20260810 18:11:59.407369 3770 leader_election.cc:302] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b626ebaba5c4745bf7321813aa45ef8, cf27092e6a0f4e0b98615b02c1d01728; no voters:
I20260810 18:11:59.407572 4576 raft_consensus.cc:2781] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:59.407677 4576 raft_consensus.cc:686] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 1 LEADER]: Becoming Leader. State: Replica: cf27092e6a0f4e0b98615b02c1d01728, State: Running, Role: LEADER
I20260810 18:11:59.407838 4576 consensus_queue.cc:237] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } }
I20260810 18:11:59.409523 3620 catalog_manager.cc:5495] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65). New cstate: current_term: 1 leader_uuid: "cf27092e6a0f4e0b98615b02c1d01728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf27092e6a0f4e0b98615b02c1d01728" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 } health_report { overall_health: UNKNOWN } } }
I20260810 18:11:59.455649 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:59.452787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:48225 (local address 127.23.65.66:35857)
0810 18:11:59.452903 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:59.452906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:59.453054 (+ 148us) server_negotiation.cc:407] Connection header received
0810 18:11:59.453167 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:59.453169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:59.453213 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:59.453275 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:59.453471 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.454024 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.454805 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.455005 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.455071 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:59.455282 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:59.455347 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:59.455391 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:59.455451 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 18:11:59.455495 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:11:59.455673 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:59.452690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:35857 (local address 127.23.65.65:48225)
0810 18:11:59.452984 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:59.452997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:59.453091 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:59.453284 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:59.453286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:59.453291 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:59.453393 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:59.453398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.454106 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.454108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:59.454720 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:59.454728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.455078 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.455080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:59.455139 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:59.455185 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:59.455400 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:59.455404 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:11:59.455444 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 18:11:59.455509 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260810 18:11:59.461256 4317 raft_consensus.cc:1261] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Refusing update from remote peer cf27092e6a0f4e0b98615b02c1d01728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:59.461290 4102 raft_consensus.cc:1261] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Refusing update from remote peer cf27092e6a0f4e0b98615b02c1d01728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:59.461666 4576 consensus_queue.cc:1035] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb12e0294d14498b8f62452b19bcdb14" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 35857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:59.461836 4458 consensus_queue.cc:1035] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b626ebaba5c4745bf7321813aa45ef8" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:59.477651 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:59.474838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:46111 (local address 127.23.65.65:46099)
0810 18:11:59.474945 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:59.474948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:59.474959 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:11:59.474994 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:59.474997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:59.475041 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:59.475107 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:59.475346 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.475891 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.476649 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.476917 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.476995 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:59.477253 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:59.477345 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:59.477416 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:59.477510 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 18:11:59.477557 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:11:59.477669 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:59.474488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:46099 (local address 127.23.65.66:46111)
0810 18:11:59.474611 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:59.474625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:59.474702 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:59.475138 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:59.475141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:59.475150 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:11:59.475249 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:59.475254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.475980 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.475983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:59.476543 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:59.476550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:59.477016 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:59.477020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:59.477108 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:59.477140 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:11:59.477432 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:11:59.477440 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:11:59.477502 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:11:59.477572 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 18:11:59.496404 3621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:59.496553 3621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:11:59.498342 3621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c0c64a9a1024e8cadbd97c030f2368a: Sending DeleteTablet for 3 replicas of tablet f71bcff179f5472ebecc3c92011b03bd
I20260810 18:11:59.498835 3841 tablet_service.cc:1514] Processing DeleteTablet for tablet f71bcff179f5472ebecc3c92011b03bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:59 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:33728
I20260810 18:11:59.498906 4068 tablet_service.cc:1514] Processing DeleteTablet for tablet f71bcff179f5472ebecc3c92011b03bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:59 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:55426
I20260810 18:11:59.498967 4292 tablet_service.cc:1514] Processing DeleteTablet for tablet f71bcff179f5472ebecc3c92011b03bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:11:59 UTC) from {username='oryx', principal='oryx/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:35374
I20260810 18:11:59.499028 4538 tablet_replica.cc:330] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728: stopping tablet replica
I20260810 18:11:59.499123 4542 tablet_replica.cc:330] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: stopping tablet replica
I20260810 18:11:59.499117 4539 tablet_replica.cc:330] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: stopping tablet replica
I20260810 18:11:59.499151 4538 raft_consensus.cc:2229] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:11:59.499222 4542 raft_consensus.cc:2229] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:59.499236 4539 raft_consensus.cc:2229] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:11:59.499313 4542 raft_consensus.cc:2258] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:59.499330 4538 raft_consensus.cc:2258] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:59.499354 4539 raft_consensus.cc:2258] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:11:59.499954 4542 ts_tablet_manager.cc:1905] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:59.500123 4538 ts_tablet_manager.cc:1905] T f71bcff179f5472ebecc3c92011b03bd P cf27092e6a0f4e0b98615b02c1d01728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:59.500399 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 3742
I20260810 18:11:59.500504 4539 ts_tablet_manager.cc:1905] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:11:59.502480 4539 ts_tablet_manager.cc:1918] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:59.502534 4539 log.cc:1196] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/f71bcff179f5472ebecc3c92011b03bd
I20260810 18:11:59.502661 4542 ts_tablet_manager.cc:1918] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:11:59.502704 4539 ts_tablet_manager.cc:1939] T f71bcff179f5472ebecc3c92011b03bd P fb12e0294d14498b8f62452b19bcdb14: Deleting consensus metadata
I20260810 18:11:59.502741 4542 log.cc:1196] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/f71bcff179f5472ebecc3c92011b03bd
I20260810 18:11:59.503374 3603 catalog_manager.cc:4841] TS fb12e0294d14498b8f62452b19bcdb14 (127.23.65.66:35857): tablet f71bcff179f5472ebecc3c92011b03bd (table test-table [id=18c9b926bed04b659e1909e63150c833]) successfully deleted
I20260810 18:11:59.504184 4542 ts_tablet_manager.cc:1939] T f71bcff179f5472ebecc3c92011b03bd P 1b626ebaba5c4745bf7321813aa45ef8: Deleting consensus metadata
I20260810 18:11:59.504871 3603 catalog_manager.cc:4841] TS 1b626ebaba5c4745bf7321813aa45ef8 (127.23.65.67:36073): tablet f71bcff179f5472ebecc3c92011b03bd (table test-table [id=18c9b926bed04b659e1909e63150c833]) successfully deleted
W20260810 18:11:59.516762 3603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:46099 (error 108)
I20260810 18:11:59.517000 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 3967
I20260810 18:11:59.520673 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:59.520172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:46099 (local address 127.0.0.1:33754)
0810 18:11:59.520482 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:59.520565 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":58,"threads_started":1}
W20260810 18:11:59.523057 3603 catalog_manager.cc:4584] TS cf27092e6a0f4e0b98615b02c1d01728 (127.23.65.65:46099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f71bcff179f5472ebecc3c92011b03bd: Network error: Client connection negotiation failed: client connection to 127.23.65.65:46099: connect: Connection refused (error 111)
I20260810 18:11:59.530395 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 4197
I20260810 18:11:59.542243 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 3579
2026-08-10T18:11:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3434 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:59 dist-test-slave-jpj1 krb5kdc[4595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:59 dist-test-slave-jpj1 krb5kdc[4595](info): set up 2 sockets
Aug 10 18:11:59 dist-test-slave-jpj1 krb5kdc[4595](info): commencing operation
krb5kdc: starting...
W20260810 18:12:00.697373 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:12:00 dist-test-slave-jpj1 krb5kdc[4595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:12:00.820688 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:45129
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:45129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:00.932869 4618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:00.933167 4618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:00.933209 4618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:00.937255 4618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:12:00.937323 4618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:00.937341 4618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:00.937357 4618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:12:00.937371 4618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:12:00.942787 4618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:45129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:45129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:00.943910 4618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:00.944958 4618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:00.953346 4630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:00.953454 4627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:00.953385 4628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:00.953583 4618 server_base.cc:1034] running on GCE node
I20260810 18:12:00.954108 4618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:00.955188 4618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:00.956331 4618 hybrid_clock.cc:648] HybridClock initialized: now 1786385520956310 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T18:12:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1421 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:12:01 dist-test-slave-jpj1 krb5kdc[4636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:12:01 dist-test-slave-jpj1 krb5kdc[4636](info): set up 2 sockets
Aug 10 18:12:01 dist-test-slave-jpj1 krb5kdc[4636](info): commencing operation
krb5kdc: starting...
W20260810 18:12:02.129431 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:12:02 dist-test-slave-jpj1 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:12:02.256958 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:34293
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:34293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:02.372836 4656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:02.373162 4656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:02.373206 4656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:02.377221 4656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:12:02.377275 4656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:02.377293 4656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:02.377308 4656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:12:02.377327 4656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:12:02.382593 4656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:34293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:34293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:02.383801 4656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:02.384899 4656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:02.393492 4664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:02.393558 4656 server_base.cc:1034] running on GCE node
W20260810 18:12:02.393500 4665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:02.393513 4668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:02.394359 4656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:02.395460 4656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:02.396622 4656 hybrid_clock.cc:648] HybridClock initialized: now 1786385522396605 us; error 29 us; skew 500 ppm
Aug 10 18:12:02 dist-test-slave-jpj1 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385522, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:02.399287 4656 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:12:02.400166 4656 webserver.cc:460] Webserver started at http://127.23.65.126:41341/ using document root <none> and password file <none>
I20260810 18:12:02.400671 4656 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:02.400730 4656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:02.400910 4656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:02.402928 4656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "a17fb4a13e904f2f9730a0241015ca72"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "080de5ff8e478eb975bada4c407ca2df"
server_key_iv: "1b8037c9c19720c482d9366b1c2ab594"
server_key_version: "encryptionkey@0"
I20260810 18:12:02.403468 4656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "a17fb4a13e904f2f9730a0241015ca72"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "080de5ff8e478eb975bada4c407ca2df"
server_key_iv: "1b8037c9c19720c482d9366b1c2ab594"
server_key_version: "encryptionkey@0"
I20260810 18:12:02.407076 4656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:12:02.409400 4675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:02.410564 4656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:12:02.410668 4656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "a17fb4a13e904f2f9730a0241015ca72"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "080de5ff8e478eb975bada4c407ca2df"
server_key_iv: "1b8037c9c19720c482d9366b1c2ab594"
server_key_version: "encryptionkey@0"
I20260810 18:12:02.410768 4656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:02.441876 4656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:02.445096 4656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:02.445294 4656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:02.452243 4656 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:34293
I20260810 18:12:02.452251 4760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:34293 every 8 connection(s)
I20260810 18:12:02.453225 4656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:12:02.454352 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 4656
I20260810 18:12:02.454505 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:12:02.454872 23813 external_mini_cluster.cc:1428] Setting key 2227cfd5a46da4935f90f0666a5688f5
I20260810 18:12:02.456620 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:12:02.463665 4761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: Bootstrap starting.
Aug 10 18:12:02 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:12:02.466604 4761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: Neither blocks nor log segments found. Creating new log.
I20260810 18:12:02.467512 4761 log.cc:824] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: Log is configured to *not* fsync() on all Append() calls
I20260810 18:12:02.469668 4761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: No bootstrap required, opened a new log
I20260810 18:12:02.471014 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:02.456261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.23.65.126:34293)
0810 18:12:02.456653 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:02.456673 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:02.456719 (+ 46us) server_negotiation.cc:407] Connection header received
0810 18:12:02.457379 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:02.457395 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:02.457718 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:02.458007 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:02.459148 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.460751 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.461967 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.462513 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.462651 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:02.466174 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:02.466180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:02.466197 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:02.466234 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:02.468084 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:02.468591 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:02.468597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:02.468602 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:02.468719 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:02.469002 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:02.469005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:02.469006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:02.469450 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:02.469656 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:02.469953 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 18:12:02.470255 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260810 18:12:02.473446 4761 raft_consensus.cc:348] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } }
I20260810 18:12:02.473690 4761 raft_consensus.cc:374] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:12:02.473752 4761 raft_consensus.cc:729] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17fb4a13e904f2f9730a0241015ca72, State: Initialized, Role: FOLLOWER
I20260810 18:12:02.474171 4761 consensus_queue.cc:260] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } }
I20260810 18:12:02.474318 4761 raft_consensus.cc:388] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:12:02.474383 4761 raft_consensus.cc:482] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:12:02.474488 4761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:12:02.475404 4761 raft_consensus.cc:504] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } }
I20260810 18:12:02.475762 4761 leader_election.cc:302] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a17fb4a13e904f2f9730a0241015ca72; no voters:
I20260810 18:12:02.476074 4761 leader_election.cc:288] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:12:02.476176 4768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:12:02.476393 4768 raft_consensus.cc:686] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 1 LEADER]: Becoming Leader. State: Replica: a17fb4a13e904f2f9730a0241015ca72, State: Running, Role: LEADER
I20260810 18:12:02.476758 4768 consensus_queue.cc:237] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } }
I20260810 18:12:02.477257 4761 sys_catalog.cc:564] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:12:02.478384 4771 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a17fb4a13e904f2f9730a0241015ca72. Latest consensus state: current_term: 1 leader_uuid: "a17fb4a13e904f2f9730a0241015ca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } } }
I20260810 18:12:02.478384 4769 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a17fb4a13e904f2f9730a0241015ca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } } }
I20260810 18:12:02.478549 4769 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: This master's current role is: LEADER
I20260810 18:12:02.478549 4771 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: This master's current role is: LEADER
I20260810 18:12:02.478952 4783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:12:02.481793 4783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:12:02.486783 4783 catalog_manager.cc:1347] Generated new cluster ID: 5548e42cb1bf49ad9b930b605b8582b5
I20260810 18:12:02.486852 4783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:12:02.513975 4783 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:12:02.515012 4783 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:12:02.523036 4783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: Generated new TSK 0
I20260810 18:12:02.523790 4783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:12:02.558080 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:34293
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:02.671092 4802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:02.671365 4802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:02.671408 4802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:02.675408 4802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:02.675458 4802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:02.675516 4802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:12:02.680962 4802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:34293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:02.681983 4802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:02.683007 4802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:02.692315 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:02.692360 4815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:02.692318 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:02.692561 4802 server_base.cc:1034] running on GCE node
I20260810 18:12:02.692938 4802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:02.693521 4802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:02.694677 4802 hybrid_clock.cc:648] HybridClock initialized: now 1786385522694658 us; error 34 us; skew 500 ppm
Aug 10 18:12:02 dist-test-slave-jpj1 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385522, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:02.697474 4802 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:12:02.698303 4802 webserver.cc:460] Webserver started at http://127.23.65.65:42969/ using document root <none> and password file <none>
I20260810 18:12:02.698832 4802 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:02.698895 4802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:02.699095 4802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:02.701133 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "631633a162814ae78a0f7f9c7a5624a8"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "c69cc27b7c142fa80c04f4359cfa8515"
server_key_iv: "b4fdcdc10dddf6708445550bae367ae9"
server_key_version: "encryptionkey@0"
I20260810 18:12:02.701679 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "631633a162814ae78a0f7f9c7a5624a8"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "c69cc27b7c142fa80c04f4359cfa8515"
server_key_iv: "b4fdcdc10dddf6708445550bae367ae9"
server_key_version: "encryptionkey@0"
I20260810 18:12:02.705466 4802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:12:02.707937 4825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:02.709111 4802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:12:02.709216 4802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "631633a162814ae78a0f7f9c7a5624a8"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "c69cc27b7c142fa80c04f4359cfa8515"
server_key_iv: "b4fdcdc10dddf6708445550bae367ae9"
server_key_version: "encryptionkey@0"
I20260810 18:12:02.709327 4802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:02.728497 4802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:02.731256 4802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:02.731427 4802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:02.732031 4802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:02.732851 4802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:02.732896 4802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:02.732942 4802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:02.732970 4802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:02.741235 4802 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:37615
I20260810 18:12:02.741290 4992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:37615 every 8 connection(s)
I20260810 18:12:02.742106 4802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:12:02.745075 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 4802
I20260810 18:12:02.745214 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:12:02.745554 23813 external_mini_cluster.cc:1428] Setting key ecb6e851563e0582262ede1fb6d0af3f
Aug 10 18:12:02 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385522, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:12:02.756673 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:02.743914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:38487 (local address 127.23.65.126:34293)
0810 18:12:02.744079 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:02.744083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:02.744976 (+ 893us) server_negotiation.cc:407] Connection header received
0810 18:12:02.746629 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:02.746632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:02.746679 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:02.746757 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:02.748298 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.748831 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.750171 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.750420 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.750516 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:02.753263 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:02.753266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:02.753268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:02.753294 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:02.754609 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:02.755104 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:02.755107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:02.755108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:02.755186 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:02.755515 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:02.755519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:02.755520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:02.755755 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:02.755852 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:02.756367 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 18:12:02.756502 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:12:02.757509 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:02.744315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:38487)
0810 18:12:02.744787 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:02.744851 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:02.746426 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:02.746948 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:02.746959 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:02.747456 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:02.748143 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:02.748162 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.748957 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.748961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:02.750046 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:02.750057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.750572 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.750576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:02.750738 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:02.751274 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:02.751291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:02.753089 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:02.754729 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:02.754735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:02.754750 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:02.755006 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:02.755271 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:02.755274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:02.755276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:02.755426 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:02.755858 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:02.755865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:02.756136 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 18:12:02.756308 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 18:12:02.756641 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
I20260810 18:12:02.758749 4993 heartbeater.cc:347] Connected to a master server at 127.23.65.126:34293
I20260810 18:12:02.759011 4993 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:02.759651 4993 heartbeater.cc:511] Master 127.23.65.126:34293 requested a full tablet report, sending...
I20260810 18:12:02.761173 4699 ts_manager.cc:194] Registered new tserver with Master: 631633a162814ae78a0f7f9c7a5624a8 (127.23.65.65:37615)
I20260810 18:12:02.762259 4699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:38487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:12:02.774718 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:34293
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:02.887130 4999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:02.887391 4999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:02.887432 4999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:02.891357 4999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:02.891409 4999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:02.891466 4999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:12:02.896524 4999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:34293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:02.897466 4999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:02.898407 4999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:02.908042 5009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:02.908046 5005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:02.908046 5012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:02.908465 4999 server_base.cc:1034] running on GCE node
I20260810 18:12:02.908963 4999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:02.909693 4999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:02.910859 4999 hybrid_clock.cc:648] HybridClock initialized: now 1786385522910838 us; error 35 us; skew 500 ppm
Aug 10 18:12:02 dist-test-slave-jpj1 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385522, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:02.913697 4999 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:12:02.914649 4999 webserver.cc:460] Webserver started at http://127.23.65.66:46563/ using document root <none> and password file <none>
I20260810 18:12:02.915210 4999 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:02.915288 4999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:02.915505 4999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:02.917582 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "fd3ca682a66e44ddb7bba701cf6a573c"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "706ff86e30d7ae752ce0e2a064591262"
server_key_iv: "0d63333902b2611bc00a3f0ccb0a6b83"
server_key_version: "encryptionkey@0"
I20260810 18:12:02.918134 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "fd3ca682a66e44ddb7bba701cf6a573c"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "706ff86e30d7ae752ce0e2a064591262"
server_key_iv: "0d63333902b2611bc00a3f0ccb0a6b83"
server_key_version: "encryptionkey@0"
I20260810 18:12:02.921794 4999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 18:12:02.924171 5021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:02.925356 4999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 18:12:02.925468 4999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "fd3ca682a66e44ddb7bba701cf6a573c"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "706ff86e30d7ae752ce0e2a064591262"
server_key_iv: "0d63333902b2611bc00a3f0ccb0a6b83"
server_key_version: "encryptionkey@0"
I20260810 18:12:02.925565 4999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:02.945871 4999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:02.949193 4999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:02.949391 4999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:02.950042 4999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:02.950938 4999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:02.950989 4999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:02.951046 4999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:02.951072 4999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:02.959576 4999 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:41159
I20260810 18:12:02.959621 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:41159 every 8 connection(s)
I20260810 18:12:02.960597 4999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:12:02.961292 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 4999
I20260810 18:12:02.961433 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:12:02.961709 23813 external_mini_cluster.cc:1428] Setting key 5a45d2441afd845f06cac88a4e733848
Aug 10 18:12:02 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385522, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:12:02.974711 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:02.962542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:38657 (local address 127.23.65.126:34293)
0810 18:12:02.962665 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:02.962669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:02.963690 (+ 1021us) server_negotiation.cc:407] Connection header received
0810 18:12:02.965369 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:02.965373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:02.965452 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:02.965546 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:02.966701 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.967216 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.968363 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.968651 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.968743 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:02.971388 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:02.971391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:02.971393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:02.971417 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:02.972737 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:02.973254 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:02.973258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:02.973259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:02.973335 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:02.973640 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:02.973644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:02.973645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:02.973844 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:02.973940 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:02.974425 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 18:12:02.974546 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
I20260810 18:12:02.975612 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:02.962947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:38657)
0810 18:12:02.963489 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:02.963580 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:02.965137 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:02.965724 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:02.965731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:02.966087 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:02.966561 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:02.966574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.967308 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.967311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:02.968238 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:02.968247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:02.968809 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:02.968822 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:02.968980 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:02.969545 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:02.969565 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:02.971235 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:02.972847 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:02.972852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:02.972868 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:02.973163 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:02.973416 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:02.973419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:02.973421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:02.973555 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:02.973938 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:02.973945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:02.974190 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 18:12:02.974361 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 18:12:02.974683 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":138,"threads_started":1}
I20260810 18:12:02.976766 5204 heartbeater.cc:347] Connected to a master server at 127.23.65.126:34293
I20260810 18:12:02.977008 5204 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:02.977497 5204 heartbeater.cc:511] Master 127.23.65.126:34293 requested a full tablet report, sending...
I20260810 18:12:02.978576 4699 ts_manager.cc:194] Registered new tserver with Master: fd3ca682a66e44ddb7bba701cf6a573c (127.23.65.66:41159)
I20260810 18:12:02.979329 4699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:38657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:12:02.991866 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:34293
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:03.105698 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:03.105984 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:03.106029 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:03.110087 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:03.110152 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:03.110210 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:12:03.115687 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:34293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:03.116776 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:03.117797 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:03.127377 5225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:03.127450 5210 server_base.cc:1034] running on GCE node
W20260810 18:12:03.127400 5227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:03.127396 5230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:03.128043 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:03.128719 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:03.129873 5210 hybrid_clock.cc:648] HybridClock initialized: now 1786385523129846 us; error 40 us; skew 500 ppm
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385523, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:03.132648 5210 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:12:03.133531 5210 webserver.cc:460] Webserver started at http://127.23.65.67:45839/ using document root <none> and password file <none>
I20260810 18:12:03.134061 5210 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:03.134131 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:03.134358 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:03.136478 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "bb2fb5a17b474a25ad050a9fb4cae02e"
format_stamp: "Formatted at 2026-08-10 18:12:03 on dist-test-slave-jpj1"
server_key: "514a21d7ba2373668f45e3a96f17123f"
server_key_iv: "15bcf51d432a5370e5a8cb90e206d88f"
server_key_version: "encryptionkey@0"
I20260810 18:12:03.137020 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "bb2fb5a17b474a25ad050a9fb4cae02e"
format_stamp: "Formatted at 2026-08-10 18:12:03 on dist-test-slave-jpj1"
server_key: "514a21d7ba2373668f45e3a96f17123f"
server_key_iv: "15bcf51d432a5370e5a8cb90e206d88f"
server_key_version: "encryptionkey@0"
I20260810 18:12:03.140906 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 18:12:03.143494 5239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.144862 5210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:12:03.144995 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "bb2fb5a17b474a25ad050a9fb4cae02e"
format_stamp: "Formatted at 2026-08-10 18:12:03 on dist-test-slave-jpj1"
server_key: "514a21d7ba2373668f45e3a96f17123f"
server_key_iv: "15bcf51d432a5370e5a8cb90e206d88f"
server_key_version: "encryptionkey@0"
I20260810 18:12:03.145110 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:03.159257 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:03.162374 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:03.162550 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:03.163129 5210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:03.164007 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:03.164057 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.164111 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:03.164136 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.172708 5210 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:34025
I20260810 18:12:03.172765 5419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:34025 every 8 connection(s)
I20260810 18:12:03.173592 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:12:03.179138 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 5210
I20260810 18:12:03.179289 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:12:03.179634 23813 external_mini_cluster.cc:1428] Setting key 7b600bfd9009594ca56fc983453d3815
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385523, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:12:03.187951 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:03.175379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:49355 (local address 127.23.65.126:34293)
0810 18:12:03.175524 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:03.175528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:03.176379 (+ 851us) server_negotiation.cc:407] Connection header received
0810 18:12:03.177771 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:03.177774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:03.177824 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:03.177895 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:03.179138 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.179938 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.181389 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.181660 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.181760 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.184256 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:03.184260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:03.184262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:03.184299 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:03.185742 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:03.186374 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:03.186377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:03.186378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:03.186456 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:03.186812 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:03.186814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:03.186815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:03.187053 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:03.187134 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:03.187705 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 18:12:03.187821 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:12:03.188900 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:03.175760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:49355)
0810 18:12:03.176231 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:03.176275 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:03.177525 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:03.178044 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:03.178051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:03.178473 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:03.178994 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.179009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.180071 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.180075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.181280 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.181290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.181816 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.181819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.181965 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.182476 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:03.182492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:03.184042 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:03.185926 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:03.185932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:03.185948 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:03.186278 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:03.186518 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:03.186520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:03.186522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:03.186731 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:03.187154 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:03.187160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:03.187433 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 18:12:03.187677 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 18:12:03.187974 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260810 18:12:03.190120 5420 heartbeater.cc:347] Connected to a master server at 127.23.65.126:34293
I20260810 18:12:03.190353 5420 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:03.190959 5420 heartbeater.cc:511] Master 127.23.65.126:34293 requested a full tablet report, sending...
I20260810 18:12:03.192042 4699 ts_manager.cc:194] Registered new tserver with Master: bb2fb5a17b474a25ad050a9fb4cae02e (127.23.65.67:34025)
I20260810 18:12:03.192655 4699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:49355
I20260810 18:12:03.193470 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab.
Entry for principal oryx/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab.
WARNING: no policy specified for oryx/127.23.65.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab.
Entry for principal oryx/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab.
WARNING: no policy specified for oryx/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab.
Entry for principal oryx/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab.
I20260810 18:12:03.271821 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 4802
I20260810 18:12:03.287710 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 4999
I20260810 18:12:03.317906 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 5210
I20260810 18:12:03.340829 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 4656
I20260810 18:12:03.354710 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:34293
--webserver_interface=127.23.65.126
--webserver_port=41341
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:34293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:03.467679 5438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:03.467983 5438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:03.468024 5438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:03.472048 5438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:12:03.472105 5438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:03.472122 5438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:03.472138 5438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:12:03.472151 5438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:12:03.477406 5438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:34293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:34293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=41341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:03.478374 5438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:03.479317 5438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:03.487438 5450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:03.487452 5446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:03.487444 5447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:03.487489 5438 server_base.cc:1034] running on GCE node
I20260810 18:12:03.488287 5438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:03.489349 5438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:03.490540 5438 hybrid_clock.cc:648] HybridClock initialized: now 1786385523490522 us; error 33 us; skew 500 ppm
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385523, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:03.493283 5438 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:12:03.494190 5438 webserver.cc:460] Webserver started at http://127.23.65.126:41341/ using document root <none> and password file <none>
I20260810 18:12:03.494771 5438 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:03.494858 5438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:03.498654 5438 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:12:03.500818 5459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.502013 5438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:12:03.502140 5438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "a17fb4a13e904f2f9730a0241015ca72"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "080de5ff8e478eb975bada4c407ca2df"
server_key_iv: "1b8037c9c19720c482d9366b1c2ab594"
server_key_version: "encryptionkey@0"
I20260810 18:12:03.502667 5438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:03.535099 5438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:03.538120 5438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:03.538306 5438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:03.545385 5438 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:34293
I20260810 18:12:03.545413 5540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:34293 every 8 connection(s)
I20260810 18:12:03.546517 5438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:12:03.550961 5543 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:12:03.551404 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 5438
I20260810 18:12:03.551870 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:37615
--local_ip_for_outbound_sockets=127.23.65.65
--tserver_master_addrs=127.23.65.126:34293
--webserver_port=42969
--webserver_interface=127.23.65.65
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.65.65
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:12:03.552829 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: Bootstrap starting.
I20260810 18:12:03.571205 5543 log.cc:824] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: Log is configured to *not* fsync() on all Append() calls
I20260810 18:12:03.577693 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:12:03.578042 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: Bootstrap complete.
I20260810 18:12:03.582139 5543 raft_consensus.cc:348] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } }
I20260810 18:12:03.582577 5543 raft_consensus.cc:729] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17fb4a13e904f2f9730a0241015ca72, State: Initialized, Role: FOLLOWER
I20260810 18:12:03.583048 5543 consensus_queue.cc:260] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } }
I20260810 18:12:03.583171 5543 raft_consensus.cc:388] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:12:03.583237 5543 raft_consensus.cc:482] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:12:03.583328 5543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:12:03.585263 5543 raft_consensus.cc:504] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } }
I20260810 18:12:03.585585 5543 leader_election.cc:302] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a17fb4a13e904f2f9730a0241015ca72; no voters:
I20260810 18:12:03.585891 5543 leader_election.cc:288] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:12:03.586000 5549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:12:03.586216 5549 raft_consensus.cc:686] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [term 2 LEADER]: Becoming Leader. State: Replica: a17fb4a13e904f2f9730a0241015ca72, State: Running, Role: LEADER
I20260810 18:12:03.586608 5549 consensus_queue.cc:237] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } }
I20260810 18:12:03.587186 5543 sys_catalog.cc:564] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:12:03.587642 5549 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a17fb4a13e904f2f9730a0241015ca72. Latest consensus state: current_term: 2 leader_uuid: "a17fb4a13e904f2f9730a0241015ca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } } }
I20260810 18:12:03.587778 5549 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: This master's current role is: LEADER
I20260810 18:12:03.588955 5550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a17fb4a13e904f2f9730a0241015ca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17fb4a13e904f2f9730a0241015ca72" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34293 } } }
I20260810 18:12:03.589125 5550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72 [sys.catalog]: This master's current role is: LEADER
I20260810 18:12:03.589242 5553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:12:03.592459 5553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:12:03.593096 5553 catalog_manager.cc:1259] Loaded cluster ID: 5548e42cb1bf49ad9b930b605b8582b5
I20260810 18:12:03.593161 5553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:12:03.595511 5553 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:12:03.596220 5553 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a17fb4a13e904f2f9730a0241015ca72: Loaded TSK: 0
I20260810 18:12:03.597496 5553 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:12:03.667729 5546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:03.668016 5546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:03.668080 5546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:03.672267 5546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:03.672343 5546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:03.672423 5546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:12:03.677739 5546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.65.keytab
--principal=oryx/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:37615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=42969
--tserver_master_addrs=127.23.65.126:34293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:03.678790 5546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:03.679811 5546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:03.689064 5576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:03.689059 5577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:03.689456 5546 server_base.cc:1034] running on GCE node
W20260810 18:12:03.689465 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:03.690042 5546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:03.690689 5546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:03.691852 5546 hybrid_clock.cc:648] HybridClock initialized: now 1786385523691820 us; error 51 us; skew 500 ppm
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385523, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:03.694695 5546 init.cc:377] Logged in from keytab as oryx/127.23.65.65@KRBTEST.COM (short username oryx)
I20260810 18:12:03.695623 5546 webserver.cc:460] Webserver started at http://127.23.65.65:42969/ using document root <none> and password file <none>
I20260810 18:12:03.696156 5546 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:03.696239 5546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:03.700165 5546 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:12:03.702287 5594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.703581 5546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:12:03.703691 5546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "631633a162814ae78a0f7f9c7a5624a8"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "c69cc27b7c142fa80c04f4359cfa8515"
server_key_iv: "b4fdcdc10dddf6708445550bae367ae9"
server_key_version: "encryptionkey@0"
I20260810 18:12:03.704212 5546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:03.724901 5546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:03.728042 5546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:03.728223 5546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:03.728829 5546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:03.729718 5546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:03.729768 5546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.729825 5546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:03.729852 5546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.738662 5546 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:37615
I20260810 18:12:03.738708 5776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:37615 every 8 connection(s)
I20260810 18:12:03.739714 5546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:12:03.749711 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 5546
I20260810 18:12:03.750157 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:41159
--local_ip_for_outbound_sockets=127.23.65.66
--tserver_master_addrs=127.23.65.126:34293
--webserver_port=46563
--webserver_interface=127.23.65.66
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.65.66
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:03.753031 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:03.741867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:34879)
0810 18:12:03.742336 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:03.742381 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:03.743676 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:03.744784 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:03.744792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:03.745201 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:03.745673 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.745687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.746961 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.746964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.747909 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.747920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.748697 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.748702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.748884 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.749474 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:03.749491 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:03.752198 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":129,"threads_started":1}
W20260810 18:12:03.753539 5777 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:34293 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.65.126:34293: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
W20260810 18:12:03.753966 5780 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:03.741752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:34879 (local address 127.23.65.126:34293)
0810 18:12:03.742267 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:03.742288 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:03.742486 (+ 198us) server_negotiation.cc:407] Connection header received
0810 18:12:03.744070 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:03.744081 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:03.744376 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:03.744676 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:03.745843 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.746822 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.748030 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.748518 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.748674 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.752543 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:34879: BlockingRecv error: recv got EOF from 127.23.65.65:34879 (error 108)
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":122,"threads_started":1}
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:03.759394 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:03.754024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:45299)
0810 18:12:03.754372 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:03.754389 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:03.754486 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:03.754729 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:03.754732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:03.754894 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:03.754992 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.754997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.755863 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.755866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.756413 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.756422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.756866 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.756870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.756956 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.757449 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:03.757459 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:03.759266 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261}
W20260810 18:12:03.759565 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:03.754105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:45299 (local address 127.23.65.126:34293)
0810 18:12:03.754390 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:03.754399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:03.754437 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:12:03.754577 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:03.754580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:03.754625 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:03.754699 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:03.755104 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.755755 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.756524 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.756764 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.756860 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.759429 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:45299: BlockingRecv error: recv got EOF from 127.23.65.65:45299 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":116,"threads_started":1}
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:03.764564 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:03.759981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:33999)
0810 18:12:03.760103 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:03.760118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:03.760199 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:03.760411 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:03.760414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:03.760556 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:03.760627 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.760632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.761326 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.761329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.761846 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.761854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.762256 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.762259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.762343 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.762766 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:03.762777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:03.764449 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:12:03.764673 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:03.760014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:33999 (local address 127.23.65.126:34293)
0810 18:12:03.760110 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:03.760114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:03.760159 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:12:03.760277 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:03.760280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:03.760321 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:03.760384 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:03.760731 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.761220 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.761953 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.762160 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.762242 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.764583 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:33999: BlockingRecv error: recv got EOF from 127.23.65.65:33999 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 18:12:03.764776 5777 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:12:03.865703 5787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:03.865947 5787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:03.866004 5787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:03.869910 5787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:03.869961 5787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:03.870034 5787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:12:03.875103 5787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.66.keytab
--principal=oryx/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:41159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=46563
--tserver_master_addrs=127.23.65.126:34293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:03.876215 5787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:03.877271 5787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:03.886775 5803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:03.886792 5787 server_base.cc:1034] running on GCE node
W20260810 18:12:03.886742 5798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:03.886736 5801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:03.887392 5787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:03.888038 5787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:03.889191 5787 hybrid_clock.cc:648] HybridClock initialized: now 1786385523889174 us; error 30 us; skew 500 ppm
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385523, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:03.891768 5787 init.cc:377] Logged in from keytab as oryx/127.23.65.66@KRBTEST.COM (short username oryx)
I20260810 18:12:03.892601 5787 webserver.cc:460] Webserver started at http://127.23.65.66:46563/ using document root <none> and password file <none>
I20260810 18:12:03.893115 5787 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:03.893199 5787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:03.896884 5787 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:12:03.898804 5812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.899979 5787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:12:03.900092 5787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "fd3ca682a66e44ddb7bba701cf6a573c"
format_stamp: "Formatted at 2026-08-10 18:12:02 on dist-test-slave-jpj1"
server_key: "706ff86e30d7ae752ce0e2a064591262"
server_key_iv: "0d63333902b2611bc00a3f0ccb0a6b83"
server_key_version: "encryptionkey@0"
I20260810 18:12:03.900604 5787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:03.909812 5787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:03.912324 5787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:03.912484 5787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:03.913007 5787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:03.913766 5787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:03.913815 5787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.913864 5787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:03.913893 5787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:03.922453 5787 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:41159
I20260810 18:12:03.922492 5997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:41159 every 8 connection(s)
I20260810 18:12:03.923511 5787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:12:03.927845 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 5787
I20260810 18:12:03.928382 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:34025
--local_ip_for_outbound_sockets=127.23.65.67
--tserver_master_addrs=127.23.65.126:34293
--webserver_port=45839
--webserver_interface=127.23.65.67
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.65.67
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:12:03.935057 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:03.925416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:46579 (local address 127.23.65.126:34293)
0810 18:12:03.925598 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:03.925603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:03.926473 (+ 870us) server_negotiation.cc:407] Connection header received
0810 18:12:03.927905 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:03.927908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:03.927972 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:03.928055 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:03.929616 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.930154 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.931290 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.931512 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.931623 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.934932 (+ 3309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:46579: BlockingRecv error: recv got EOF from 127.23.65.66:46579 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 18:12:03.935703 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:03.925826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:46579)
0810 18:12:03.926309 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:03.926357 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:03.927667 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:03.928268 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:03.928278 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:03.928771 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:03.929443 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.929462 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.930297 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.930300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.931164 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.931174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.931669 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.931673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.931846 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.932411 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:03.932427 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:03.934784 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
W20260810 18:12:03.936266 5999 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:34293 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.65.126:34293: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:03.942035 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:03.936786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:44019)
0810 18:12:03.936974 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:03.936989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:03.937069 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:03.937301 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:03.937304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:03.937479 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:03.937608 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.937616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.938371 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.938375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.939212 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.939224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.939663 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.939667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.939764 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.940205 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:03.940221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:03.941941 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260810 18:12:03.942335 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:03.936989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:44019 (local address 127.23.65.126:34293)
0810 18:12:03.937100 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:03.937103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:03.937114 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:12:03.937147 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:03.937149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:03.937188 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:03.937266 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:03.937766 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.938259 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.939323 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.939528 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.939648 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.942206 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:44019: BlockingRecv error: recv got EOF from 127.23.65.66:44019 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:03 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:03.946920 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:03.942497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:56829)
0810 18:12:03.942646 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:03.942658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:03.942719 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:03.942951 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:03.942954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:03.943086 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:03.943160 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.943165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.943867 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.943871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.944375 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:03.944382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.944777 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.944781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:03.944855 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.945266 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:03.945276 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:03.946800 (+ 1524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 18:12:03.946993 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:03.942668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:56829 (local address 127.23.65.126:34293)
0810 18:12:03.942787 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:03.942791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:03.942802 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:12:03.942832 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:03.942834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:03.942865 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:03.942926 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:03.943260 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.943778 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.944474 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:03.944684 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:03.944766 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:03.946907 (+ 2141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:56829: BlockingRecv error: recv got EOF from 127.23.65.66:56829 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 18:12:03.947218 5999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:12:04.043603 6002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:04.043856 6002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:04.043898 6002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:04.047835 6002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:04.047909 6002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:04.047966 6002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:12:04.053301 6002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/krb5kdc/oryx_127.23.65.67.keytab
--principal=oryx/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:34025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=45839
--tserver_master_addrs=127.23.65.126:34293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:04.054373 6002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:04.055385 6002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:04.064682 6011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:04.064683 6010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:04.064741 6013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:04.064891 6002 server_base.cc:1034] running on GCE node
I20260810 18:12:04.065433 6002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:04.066107 6002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:04.067277 6002 hybrid_clock.cc:648] HybridClock initialized: now 1786385524067243 us; error 47 us; skew 500 ppm
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385524, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:04.070001 6002 init.cc:377] Logged in from keytab as oryx/127.23.65.67@KRBTEST.COM (short username oryx)
I20260810 18:12:04.070859 6002 webserver.cc:460] Webserver started at http://127.23.65.67:45839/ using document root <none> and password file <none>
I20260810 18:12:04.071374 6002 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:04.071448 6002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:04.075289 6002 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:12:04.077330 6021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:04.078516 6002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:12:04.078629 6002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "bb2fb5a17b474a25ad050a9fb4cae02e"
format_stamp: "Formatted at 2026-08-10 18:12:03 on dist-test-slave-jpj1"
server_key: "514a21d7ba2373668f45e3a96f17123f"
server_key_iv: "15bcf51d432a5370e5a8cb90e206d88f"
server_key_version: "encryptionkey@0"
I20260810 18:12:04.079172 6002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:04.093451 6002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:04.096494 6002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:04.096683 6002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:04.097280 6002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:04.098129 6002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:04.098178 6002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:04.098233 6002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:04.098260 6002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:04.106930 6002 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:34025
I20260810 18:12:04.106958 6186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:34025 every 8 connection(s)
I20260810 18:12:04.108011 6002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:12:04.115784 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 6002
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:12:04.118748 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:04.109819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:56833 (local address 127.23.65.126:34293)
0810 18:12:04.109960 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:04.109965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:04.110843 (+ 878us) server_negotiation.cc:407] Connection header received
0810 18:12:04.112187 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:04.112190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:04.112236 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:04.112296 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:04.113413 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.113919 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.114969 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.115173 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.115240 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.118646 (+ 3406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:56833: BlockingRecv error: recv got EOF from 127.23.65.67:56833 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:12:04.119441 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:04.110240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:56833)
0810 18:12:04.110694 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:04.110737 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:04.111980 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:04.112451 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:04.112457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:04.112797 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:04.113292 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.113306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.114009 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.114012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.114881 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.114890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.115278 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.115281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.115415 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.116009 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:04.116026 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:04.118541 (+ 2515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
W20260810 18:12:04.119992 6187 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:34293 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.65.126:34293: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:04.125550 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:04.116847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.23.65.126:34293)
0810 18:12:04.116978 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:04.116982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:04.116995 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:12:04.117031 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:04.117034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:04.117080 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:04.117149 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:04.117946 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.118697 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.119600 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.119883 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.119976 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.121596 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:04.121601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:04.121615 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:04.121641 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:04.123472 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:04.123915 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:04.123922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:04.123929 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:04.124100 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:04.124420 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:04.124423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:04.124424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:04.124790 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:04.124954 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:04.125241 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 18:12:04.125440 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:12:04.125772 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:04.120385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:49397)
0810 18:12:04.120493 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:04.120509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:04.120579 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:04.120818 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:04.120821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:04.120943 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:04.121032 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.121037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.121864 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.121867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.122523 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.122534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.123108 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.123111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.123187 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.123672 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:04.123684 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:04.125663 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 18:12:04.126194 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:04.120480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:49397 (local address 127.23.65.126:34293)
0810 18:12:04.120576 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:04.120580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:04.120592 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:12:04.120657 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:04.120659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:04.120707 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:04.120774 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:04.121134 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.121765 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.122654 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.122969 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.123083 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.126117 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:49397: BlockingRecv error: recv got EOF from 127.23.65.67:49397 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:04.130884 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:04.126222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:60903)
0810 18:12:04.126374 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:04.126387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:04.126455 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:04.126715 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:04.126717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:04.126835 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:04.126905 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.126910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.127731 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.127734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.128306 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.128316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.128737 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.128740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.128807 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.129206 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:04.129217 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:04.130767 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260810 18:12:04.131001 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:04.126402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:60903 (local address 127.23.65.126:34293)
0810 18:12:04.126520 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:04.126524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:04.126537 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:12:04.126569 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:04.126572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:04.126609 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:04.126682 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:04.126998 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.127631 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.128419 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.128637 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.128709 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.130898 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:60903: BlockingRecv error: recv got EOF from 127.23.65.67:60903 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 18:12:04.131145 6187 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:04.770466 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:04.765357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:35065)
0810 18:12:04.765624 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:04.765637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:04.765720 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:04.765935 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:04.765937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:04.766111 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:04.766228 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.766233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.766913 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.766915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.767482 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.767489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.768014 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.768018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.768112 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.768578 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:04.768588 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:04.770307 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
W20260810 18:12:04.770560 6195 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:04.765437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:35065 (local address 127.23.65.126:34293)
0810 18:12:04.765689 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:04.765693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:04.765706 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:12:04.765794 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:04.765796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:04.765840 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:04.765913 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:04.766311 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.766839 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.767660 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.767904 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.767996 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.770444 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:35065: BlockingRecv error: recv got EOF from 127.23.65.65:35065 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:04 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:04.952924 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:04.947855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:39121)
0810 18:12:04.948101 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:04.948114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:04.948213 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:04.948529 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:04.948531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:04.948709 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:04.948815 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.948820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.949550 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.949552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.950127 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:04.950134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.950548 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.950551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:04.950615 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.951066 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:04.951077 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:04.952792 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
W20260810 18:12:04.953050 6195 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:04.947905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:39121 (local address 127.23.65.126:34293)
0810 18:12:04.948046 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:04.948049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:04.948176 (+ 127us) server_negotiation.cc:407] Connection header received
0810 18:12:04.948281 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:04.948283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:04.948330 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:04.948507 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:04.948933 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.949462 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.950233 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:04.950453 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:04.950530 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:04.952930 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:39121: BlockingRecv error: recv got EOF from 127.23.65.66:39121 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 18:12:05 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:05 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:05.137346 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:05.131779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:59225)
0810 18:12:05.132054 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:05.132070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:05.132182 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:05.132438 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:05.132440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:05.132648 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:05.132751 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:05.132756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.133505 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.133510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:05.134109 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:05.134118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.134569 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.134572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:05.134669 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:05.135190 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:05.135202 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:05.137201 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
W20260810 18:12:05.137481 6195 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:05.131895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:59225 (local address 127.23.65.126:34293)
0810 18:12:05.132038 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:05.132042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:05.132127 (+ 85us) server_negotiation.cc:407] Connection header received
0810 18:12:05.132270 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:05.132273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:05.132328 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:05.132394 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:05.132868 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.133397 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.134230 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.134460 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.134551 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:05.137365 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:59225: BlockingRecv error: recv got EOF from 127.23.65.67:59225 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 18:12:05 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:05 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:05.776921 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:05.771253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:58553)
0810 18:12:05.771579 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:05.771594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:05.771692 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:05.772050 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:05.772053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:05.772252 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:05.772390 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:05.772399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.773224 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.773227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:05.773803 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:05.773811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.774258 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.774260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:05.774325 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:05.774819 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:05.774831 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:05.776752 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
W20260810 18:12:05.777091 6200 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:05.771336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:58553 (local address 127.23.65.126:34293)
0810 18:12:05.771649 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:05.771653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:05.771671 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:12:05.771868 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:05.771871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:05.771916 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:05.772006 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:05.772501 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.773126 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.773922 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.774162 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.774243 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:05.776936 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:58553: BlockingRecv error: recv got EOF from 127.23.65.65:58553 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":60,"threads_started":1}
Aug 10 18:12:05 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:05 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:05.958817 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:05.953639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:56485)
0810 18:12:05.953915 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:05.953929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:05.954036 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:05.954239 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:05.954241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:05.954431 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:05.954536 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:05.954542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.955311 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.955321 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:05.955910 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:05.955920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.956356 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.956360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:05.956441 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:05.956927 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:05.956940 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:05.958672 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
W20260810 18:12:05.958930 6200 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:05.953726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:56485 (local address 127.23.65.126:34293)
0810 18:12:05.953885 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:05.953890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:05.953983 (+ 93us) server_negotiation.cc:407] Connection header received
0810 18:12:05.954100 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:05.954103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:05.954147 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:05.954210 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:05.954665 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.955199 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.956026 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:05.956245 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:05.956333 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:05.958806 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:56485: BlockingRecv error: recv got EOF from 127.23.65.66:56485 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 18:12:06 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:06 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:06.146464 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:06.138289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:46223)
0810 18:12:06.138568 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:06.138585 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:06.138739 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:06.139002 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:06.139004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:06.139239 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:06.139352 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:06.139358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.140341 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.140345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:06.141049 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:06.141061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.141474 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.141477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:06.141535 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:06.144655 (+ 3120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:06.144668 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:06.146324 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
W20260810 18:12:06.146576 6200 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:06.138501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:46223 (local address 127.23.65.126:34293)
0810 18:12:06.138628 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:06.138633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:06.138649 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:12:06.138843 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:06.138845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:06.138893 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:06.138959 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:06.139482 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.140188 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.141166 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.141388 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.141463 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:06.146471 (+ 5008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:46223: BlockingRecv error: recv got EOF from 127.23.65.67:46223 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 18:12:06 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:06 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:06.782768 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:06.777717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:47705)
0810 18:12:06.778017 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:06.778030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:06.778122 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:06.778331 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:06.778333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:06.778514 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:06.778613 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:06.778618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.779303 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.779305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:06.779909 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:06.779917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.780282 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.780284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:06.780341 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:06.780796 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:06.780807 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:06.782608 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
W20260810 18:12:06.782892 6211 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:06.777849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:47705 (local address 127.23.65.126:34293)
0810 18:12:06.778091 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:06.778094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:06.778107 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:12:06.778203 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:06.778206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:06.778250 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:06.778332 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:06.778700 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.779225 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.779988 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.780204 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.780274 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:06.782753 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:47705: BlockingRecv error: recv got EOF from 127.23.65.65:47705 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
Aug 10 18:12:06 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:06 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:06.964886 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:06.959577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:51183)
0810 18:12:06.959873 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:06.959890 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:06.960018 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:06.960271 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:06.960274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:06.960488 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:06.960596 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:06.960604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.961350 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.961353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:06.961911 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:06.961920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.962359 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.962363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:06.962449 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:06.962914 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:06.962925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:06.964723 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
W20260810 18:12:06.964968 6211 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:06.959739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:51183 (local address 127.23.65.126:34293)
0810 18:12:06.959887 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:06.959892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:06.959965 (+ 73us) server_negotiation.cc:407] Connection header received
0810 18:12:06.960111 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:06.960114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:06.960166 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:06.960232 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:06.960718 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.961246 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.962025 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:06.962253 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:06.962341 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:06.964868 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:51183: BlockingRecv error: recv got EOF from 127.23.65.66:51183 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 18:12:07 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:07 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:07.152201 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:07.147237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:54755)
0810 18:12:07.147507 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:07.147520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:07.147639 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:07.147907 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:07.147909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:07.148091 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:07.148197 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:07.148202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.148880 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.148882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:07.149438 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:07.149445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.149804 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.149806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:07.149866 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:07.150332 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:07.150344 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:07.152057 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
W20260810 18:12:07.152305 6211 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:07.147327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:54755 (local address 127.23.65.126:34293)
0810 18:12:07.147494 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:07.147497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:07.147597 (+ 100us) server_negotiation.cc:407] Connection header received
0810 18:12:07.147731 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:07.147733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:07.147783 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:07.147863 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:07.148296 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.148804 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.149517 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.149727 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.149793 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:07.152194 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:54755: BlockingRecv error: recv got EOF from 127.23.65.67:54755 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 18:12:07 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:07 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:07.788713 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:07.783582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:44443)
0810 18:12:07.783892 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:07.783905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:07.784006 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:07.784220 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:07.784222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:07.784401 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:07.784501 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:07.784506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.785202 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.785205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:07.785777 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:07.785786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.786148 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.786151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:07.786208 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:07.786645 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:07.786656 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:07.788544 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":121,"threads_started":1}
W20260810 18:12:07.788825 6217 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:07.783629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:44443 (local address 127.23.65.126:34293)
0810 18:12:07.783897 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:07.783901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:07.783965 (+ 64us) server_negotiation.cc:407] Connection header received
0810 18:12:07.784075 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:07.784077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:07.784122 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:07.784202 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:07.784586 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.785114 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.785862 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.786073 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.786139 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:07.788692 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:44443: BlockingRecv error: recv got EOF from 127.23.65.65:44443 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
Aug 10 18:12:07 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:07 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:07.970976 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:07.965642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:41999)
0810 18:12:07.965935 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:07.965950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:07.966071 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:07.966337 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:07.966340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:07.966557 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:07.966665 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:07.966672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.967423 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.967427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:07.968043 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:07.968054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.968508 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.968512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:07.968596 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:07.969046 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:07.969057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:07.970854 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
W20260810 18:12:07.971076 6217 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:07.965744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:41999 (local address 127.23.65.126:34293)
0810 18:12:07.965910 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:07.965915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:07.966020 (+ 105us) server_negotiation.cc:407] Connection header received
0810 18:12:07.966175 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:07.966178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:07.966233 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:07.966301 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:07.966785 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.967316 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.968170 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:07.968404 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:07.968491 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:07.970966 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:41999: BlockingRecv error: recv got EOF from 127.23.65.66:41999 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 18:12:08 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:08 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:08.158175 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:08.152934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:33359)
0810 18:12:08.153186 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:08.153199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:08.153301 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:08.153563 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:08.153566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:08.153759 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:08.153867 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:08.153873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.154592 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.154595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:08.155186 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:08.155196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.155654 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.155658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:08.155733 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:08.156197 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:08.156208 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:08.158031 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
W20260810 18:12:08.158293 6217 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:08.153016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:33359 (local address 127.23.65.126:34293)
0810 18:12:08.153182 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:08.153185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:08.153264 (+ 79us) server_negotiation.cc:407] Connection header received
0810 18:12:08.153373 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:08.153375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:08.153423 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:08.153488 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:08.153962 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.154476 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.155280 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.155483 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.155575 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:08.158187 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:33359: BlockingRecv error: recv got EOF from 127.23.65.67:33359 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 18:12:08 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:08 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:08.795042 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:08.789519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:54999)
0810 18:12:08.789847 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:08.789862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:08.789959 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:08.790235 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:08.790238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:08.790531 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:08.790661 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:08.790671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.791512 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.791516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:08.792206 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:08.792219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.792651 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.792653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:08.792715 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:08.793165 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:08.793176 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:08.794901 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":124,"threads_started":1}
W20260810 18:12:08.795156 6221 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:08.789555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:54999 (local address 127.23.65.126:34293)
0810 18:12:08.789932 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:08.789937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:08.789953 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:12:08.790043 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:08.790046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:08.790096 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:08.790186 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:08.790778 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.791389 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.792334 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.792561 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.792640 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:08.795031 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:54999: BlockingRecv error: recv got EOF from 127.23.65.65:54999 (error 108)
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":84,"threads_started":1}
Aug 10 18:12:08 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:08 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:08.977370 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:08.971968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:44873)
0810 18:12:08.972245 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:08.972258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:08.972359 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:08.972566 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:08.972568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:08.972760 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:08.972879 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:08.972885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.973593 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.973595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:08.974177 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:08.974184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.974589 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.974594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:08.974682 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:08.975239 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:08.975254 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:08.977236 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
W20260810 18:12:08.977481 6221 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:08.972108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:44873 (local address 127.23.65.126:34293)
0810 18:12:08.972247 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:08.972251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:08.972319 (+ 68us) server_negotiation.cc:407] Connection header received
0810 18:12:08.972430 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:08.972432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:08.972475 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:08.972542 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:08.972975 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.973509 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.974259 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:08.974480 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:08.974550 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:08.977377 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:44873: BlockingRecv error: recv got EOF from 127.23.65.66:44873 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 18:12:09 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:09 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:09.167491 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:09.158993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:44205)
0810 18:12:09.159260 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:09.159274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:09.159373 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:09.159675 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:09.159677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:09.159881 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:09.159986 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:09.159992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.160665 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.160667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:09.161320 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:09.161331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.161862 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.161865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:09.161947 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:09.165453 (+ 3506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:09.165466 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:09.167333 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
W20260810 18:12:09.167614 6221 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:09.159117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:44205 (local address 127.23.65.126:34293)
0810 18:12:09.159244 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:09.159248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:09.159339 (+ 91us) server_negotiation.cc:407] Connection header received
0810 18:12:09.159493 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:09.159496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:09.159567 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:09.159642 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:09.160075 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.160585 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.161438 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.161734 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.161828 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:09.167481 (+ 5653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:44205: BlockingRecv error: recv got EOF from 127.23.65.67:44205 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 18:12:09 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:09 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:09.801369 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:09.795858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:55665)
0810 18:12:09.796323 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:09.796339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:09.796445 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:09.796725 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:09.796727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:09.796966 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:09.797085 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:09.797092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.797899 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.797902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:09.798465 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:09.798473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.798923 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.798926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:09.799007 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:09.799464 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:09.799475 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:09.801195 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
W20260810 18:12:09.801457 6228 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:09.795964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:55665 (local address 127.23.65.126:34293)
0810 18:12:09.796276 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:09.796281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:09.796401 (+ 120us) server_negotiation.cc:407] Connection header received
0810 18:12:09.796530 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:09.796533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:09.796590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:09.796681 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:09.797205 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.797783 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.798583 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.798812 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.798906 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:09.801340 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:55665: BlockingRecv error: recv got EOF from 127.23.65.65:55665 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":81,"threads_started":1}
Aug 10 18:12:09 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:09 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:09.983662 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:09.978247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:45619)
0810 18:12:09.978554 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:09.978571 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:09.978687 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:09.978949 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:09.978951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:09.979194 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:09.979326 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:09.979333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.980209 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.980212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:09.980765 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:09.980783 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.981144 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.981146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:09.981203 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:09.981654 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:09.981665 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:09.983499 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
W20260810 18:12:09.983772 6228 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:09.978316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:45619 (local address 127.23.65.126:34293)
0810 18:12:09.978465 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:09.978470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:09.978657 (+ 187us) server_negotiation.cc:407] Connection header received
0810 18:12:09.978778 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:09.978780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:09.978834 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:09.978911 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:09.979449 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.980109 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.980862 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:09.981068 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:09.981134 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:09.983669 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:45619: BlockingRecv error: recv got EOF from 127.23.65.66:45619 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 18:12:10 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:10 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:10.173633 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:10.168311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:48881)
0810 18:12:10.168635 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:10.168651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:10.168770 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:10.169014 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:10.169017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:10.169243 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:10.169343 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:10.169349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.170143 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.170147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:10.170700 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:10.170708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.171139 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.171142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:10.171230 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:10.171722 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:10.171733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:10.173476 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":91,"threads_started":1}
W20260810 18:12:10.173775 6228 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:10.168408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:48881 (local address 127.23.65.126:34293)
0810 18:12:10.168610 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:10.168615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:10.168723 (+ 108us) server_negotiation.cc:407] Connection header received
0810 18:12:10.168857 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:10.168860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:10.168907 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:10.168973 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:10.169461 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.170025 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.170813 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.171030 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.171115 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:10.173660 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:48881: BlockingRecv error: recv got EOF from 127.23.65.67:48881 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 18:12:10 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:10 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:10.807559 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:10.802154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:33217)
0810 18:12:10.802569 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:10.802582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:10.802677 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:10.802901 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:10.802904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:10.803090 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:10.803189 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:10.803194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.804012 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.804015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:10.804579 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:10.804589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.805042 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.805049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:10.805117 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:10.805593 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:10.805604 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:10.807368 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":73,"threads_started":1}
W20260810 18:12:10.807648 6237 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:10.802216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:33217 (local address 127.23.65.126:34293)
0810 18:12:10.802494 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:10.802499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:10.802642 (+ 143us) server_negotiation.cc:407] Connection header received
0810 18:12:10.802750 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:10.802752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:10.802806 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:10.802870 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:10.803283 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.803911 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.804670 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.804944 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.805019 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:10.807492 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:33217: BlockingRecv error: recv got EOF from 127.23.65.65:33217 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":122,"threads_started":1}
Aug 10 18:12:10 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:10 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:10.990180 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:10.984393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:51285)
0810 18:12:10.984771 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:10.984784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:10.984891 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:10.985190 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:10.985193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:10.985426 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:10.985546 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:10.985556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.986361 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.986365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:10.986968 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:10.986979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.987425 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.987429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:10.987527 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:10.988039 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:10.988052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:10.990012 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":96,"threads_started":1}
W20260810 18:12:10.990250 6237 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:10.984471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:51285 (local address 127.23.65.126:34293)
0810 18:12:10.984625 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:10.984630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:10.984850 (+ 220us) server_negotiation.cc:407] Connection header received
0810 18:12:10.984988 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:10.984992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:10.985049 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:10.985121 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:10.985661 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.986253 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.987074 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:10.987316 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:10.987410 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:10.990151 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:51285: BlockingRecv error: recv got EOF from 127.23.65.66:51285 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 18:12:11 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:11 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:11.182528 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:11.174486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:57565)
0810 18:12:11.174868 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:11.174881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:11.174992 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:11.175244 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:11.175246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:11.175471 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:11.175616 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:11.175624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.176374 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.176377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:11.176927 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:11.176935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.177368 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.177371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:11.177454 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:11.180473 (+ 3019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:11.180489 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:11.182411 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":122,"threads_started":1}
W20260810 18:12:11.182662 6237 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:11.174571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:57565 (local address 127.23.65.126:34293)
0810 18:12:11.174739 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:11.174743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:11.174956 (+ 213us) server_negotiation.cc:407] Connection header received
0810 18:12:11.175080 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:11.175083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:11.175137 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:11.175201 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:11.175724 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.176266 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.177038 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.177257 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.177350 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:11.182543 (+ 5193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:57565: BlockingRecv error: recv got EOF from 127.23.65.67:57565 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 18:12:11 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:11 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:11.813746 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:11.808286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:41069)
0810 18:12:11.808574 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:11.808588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:11.808686 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:11.808927 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:11.808929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:11.809123 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:11.809237 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:11.809244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.810063 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.810065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:11.810657 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:11.810667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.811183 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.811186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:11.811260 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:11.811799 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:11.811811 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:11.813620 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
W20260810 18:12:11.813910 6243 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:11.808421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:41069 (local address 127.23.65.126:34293)
0810 18:12:11.808637 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:11.808641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:11.808653 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:12:11.808771 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:11.808773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:11.808820 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:11.808893 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:11.809352 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.809964 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.810759 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.810990 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.811066 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:11.813780 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:41069: BlockingRecv error: recv got EOF from 127.23.65.65:41069 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
Aug 10 18:12:11 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:11 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:11.996236 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:11.990923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:54207)
0810 18:12:11.991194 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:11.991207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:11.991325 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:11.991587 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:11.991590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:11.991795 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:11.991892 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:11.991898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.992778 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.992782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:11.993375 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:11.993384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.993808 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.993811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:11.993892 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:11.994351 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:11.994362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:11.996103 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
W20260810 18:12:11.996446 6243 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:11.990955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:54207 (local address 127.23.65.126:34293)
0810 18:12:11.991087 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:11.991092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:11.991277 (+ 185us) server_negotiation.cc:407] Connection header received
0810 18:12:11.991413 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:11.991416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:11.991463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:11.991527 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:11.992003 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.992672 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.993486 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:11.993705 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:11.993800 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:11.996306 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:54207: BlockingRecv error: recv got EOF from 127.23.65.66:54207 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 18:12:12 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:12 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:12.188963 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:12.183245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:53821)
0810 18:12:12.183529 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:12.183565 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:12.183666 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:12.183899 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:12.183901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:12.184083 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:12.184183 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:12.184189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.184882 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.184884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:12.185632 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:12.185645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.186165 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.186168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:12.186244 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:12.186837 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:12.186849 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:12.188808 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
W20260810 18:12:12.189059 6243 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:12.183320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:53821 (local address 127.23.65.126:34293)
0810 18:12:12.183454 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:12.183458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:12.183625 (+ 167us) server_negotiation.cc:407] Connection header received
0810 18:12:12.183736 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:12.183738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:12.183787 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:12.183874 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:12.184269 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.184786 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.185750 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.186050 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.186148 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:12.188957 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:53821: BlockingRecv error: recv got EOF from 127.23.65.67:53821 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 18:12:12 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:12 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:12.819964 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:12.814489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:59539)
0810 18:12:12.814761 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:12.814775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:12.814856 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:12.815069 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:12.815071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:12.815264 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:12.815380 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:12.815385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.816100 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.816102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:12.816676 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:12.816685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.817178 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.817181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:12.817252 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:12.817728 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:12.817742 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:12.819808 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
W20260810 18:12:12.820087 6249 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:12.814568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:59539 (local address 127.23.65.126:34293)
0810 18:12:12.814794 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:12.814797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:12.814836 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:12:12.814929 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:12.814932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:12.814976 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:12.815043 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:12.815464 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.816023 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.816800 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.817037 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.817132 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:12.819951 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:59539: BlockingRecv error: recv got EOF from 127.23.65.65:59539 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
Aug 10 18:12:12 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:12 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:13.002593 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:12.997059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:46431)
0810 18:12:12.997393 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:12.997405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:12.997505 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:12.997837 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:12.997840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:12.998026 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:12.998128 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:12.998134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.998912 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.998915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:12.999502 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:12.999511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.999986 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.999989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:13.000058 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:13.000559 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:13.000571 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:13.002463 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":92,"threads_started":1}
W20260810 18:12:13.002736 6249 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:12.997184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:46431 (local address 127.23.65.126:34293)
0810 18:12:12.997324 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:12.997328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:12.997468 (+ 140us) server_negotiation.cc:407] Connection header received
0810 18:12:12.997567 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:12.997569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:12.997645 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:12.997729 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:12.998239 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.998783 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.999657 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:12.999878 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:12.999957 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:13.002617 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:46431: BlockingRecv error: recv got EOF from 127.23.65.66:46431 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 18:12:13 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:13 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:13.195371 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:13.189740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:41741)
0810 18:12:13.190025 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:13.190039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:13.190156 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:13.190413 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:13.190416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:13.190619 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:13.190721 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:13.190728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:13.191525 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:13.191528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:13.192283 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:13.192295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:13.192804 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:13.192808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:13.192889 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:13.193399 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:13.193411 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:13.195217 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
W20260810 18:12:13.195458 6249 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:13.189809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:41741 (local address 127.23.65.126:34293)
0810 18:12:13.189959 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:13.189964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:13.190110 (+ 146us) server_negotiation.cc:407] Connection header received
0810 18:12:13.190243 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:13.190247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:13.190299 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:13.190369 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:13.190844 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:13.191405 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:13.192405 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:13.192690 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:13.192785 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:13.195355 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:41741: BlockingRecv error: recv got EOF from 127.23.65.67:41741 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 18:12:13 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:13 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:13.825947 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:13.820729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:35617)
0810 18:12:13.820995 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:13.821009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:13.821094 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:13.821314 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:13.821317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:13.821503 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:13.821606 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:13.821612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:13.822485 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:13.822488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:13.823069 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:13.823079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:13.823463 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:13.823466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:13.823556 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:13.824020 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:13.824032 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:13.825792 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
W20260810 18:12:13.826036 6254 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:13.820818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:35617 (local address 127.23.65.126:34293)
0810 18:12:13.821063 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:13.821066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:13.821080 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:12:13.821169 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:13.821171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:13.821216 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:13.821300 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:13.821693 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:13.822356 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:13.823159 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:13.823380 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:13.823449 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:13.825922 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:35617: BlockingRecv error: recv got EOF from 127.23.65.65:35617 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
Aug 10 18:12:14 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:14 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:14.008878 6255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:14.003370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:40305)
0810 18:12:14.003655 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:14.003668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:14.003790 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:14.004015 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:14.004017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:14.004202 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:14.004318 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:14.004325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.005114 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.005117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:14.005694 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:14.005705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.006105 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.006108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:14.006178 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:14.006737 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:14.006750 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:14.008699 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":73,"threads_started":1}
W20260810 18:12:14.008916 6254 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:14.003449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:40305 (local address 127.23.65.126:34293)
0810 18:12:14.003610 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:14.003615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:14.003743 (+ 128us) server_negotiation.cc:407] Connection header received
0810 18:12:14.003878 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:14.003880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:14.003928 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:14.003989 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:14.004403 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.005009 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.005785 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.006009 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.006087 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:14.008815 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:40305: BlockingRecv error: recv got EOF from 127.23.65.66:40305 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 18:12:14 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:14 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:14.201323 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:14.196122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:38849)
0810 18:12:14.196397 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:14.196412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:14.196531 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:14.196768 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:14.196771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:14.196969 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:14.197068 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:14.197074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.197836 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.197840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:14.198398 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:14.198407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.198832 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.198836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:14.198919 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:14.199363 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:14.199374 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:14.201175 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
W20260810 18:12:14.201416 6254 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:14.196273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:38849 (local address 127.23.65.126:34293)
0810 18:12:14.196414 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:14.196418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:14.196487 (+ 69us) server_negotiation.cc:407] Connection header received
0810 18:12:14.196614 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:14.196617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:14.196666 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:14.196732 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:14.197189 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.197728 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.198507 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.198731 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.198817 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:14.201316 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:38849: BlockingRecv error: recv got EOF from 127.23.65.67:38849 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 18:12:14 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:14 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:14.832544 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:14.826930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:36501)
0810 18:12:14.827211 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:14.827224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:14.827329 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:14.827631 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:14.827634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:14.827838 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:14.827942 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:14.827949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.828750 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.828753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:14.829351 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:14.829361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.829770 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.829773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:14.829837 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:14.830357 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:14.830370 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:14.832389 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
W20260810 18:12:14.832651 6261 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:14.827076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:36501 (local address 127.23.65.126:34293)
0810 18:12:14.827325 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:14.827328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:14.827341 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:12:14.827414 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:14.827417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:14.827470 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:14.827608 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:14.828037 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.828617 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.829451 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:14.829679 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:14.829749 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:14.832535 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:36501: BlockingRecv error: recv got EOF from 127.23.65.65:36501 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
Aug 10 18:12:15 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:15 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:15.015312 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:15.009744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:59623)
0810 18:12:15.010031 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:15.010045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:15.010160 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:15.010377 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:15.010379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:15.010560 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:15.010662 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:15.010668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.011508 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.011511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:15.012327 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:15.012339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.012888 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.012891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:15.012958 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:15.013410 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:15.013420 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:15.015169 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
W20260810 18:12:15.015401 6261 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:15.009789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:59623 (local address 127.23.65.126:34293)
0810 18:12:15.009963 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:15.009968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:15.010112 (+ 144us) server_negotiation.cc:407] Connection header received
0810 18:12:15.010239 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:15.010241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:15.010287 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:15.010353 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:15.010824 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.011393 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.012458 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.012784 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.012868 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:15.015295 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:59623: BlockingRecv error: recv got EOF from 127.23.65.66:59623 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 18:12:15 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:15 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:15.207367 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:15.202067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:44329)
0810 18:12:15.202353 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:15.202365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:15.202473 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:15.202718 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:15.202721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:15.202926 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:15.203026 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:15.203032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.203875 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.203879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:15.204431 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:15.204440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.204935 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.204939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:15.205020 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:15.205471 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:15.205481 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:15.207188 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
W20260810 18:12:15.207432 6261 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:15.202141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:44329 (local address 127.23.65.126:34293)
0810 18:12:15.202297 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:15.202301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:15.202428 (+ 127us) server_negotiation.cc:407] Connection header received
0810 18:12:15.202570 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:15.202573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:15.202624 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:15.202683 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:15.203134 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.203764 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.204538 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.204812 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.204903 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:15.207330 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:44329: BlockingRecv error: recv got EOF from 127.23.65.67:44329 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 18:12:15 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:15 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:15.838769 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:15.833366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:39991)
0810 18:12:15.833660 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:15.833673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:15.833760 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:15.833982 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:15.833984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:15.834171 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:15.834274 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:15.834279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.834976 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.834978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:15.835580 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:15.835590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.835967 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.835969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:15.836028 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:15.836542 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:15.836556 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:15.838622 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
W20260810 18:12:15.838899 6267 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:15.833473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:39991 (local address 127.23.65.126:34293)
0810 18:12:15.833699 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:15.833702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:15.833715 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:12:15.833836 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:15.833838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:15.833880 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:15.833954 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:15.834370 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.834898 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.835661 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:15.835886 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:15.835957 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:15.838776 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:39991: BlockingRecv error: recv got EOF from 127.23.65.65:39991 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
Aug 10 18:12:16 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:16 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:16.021920 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:16.016197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:40151)
0810 18:12:16.016524 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:16.016540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:16.016667 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:16.016930 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:16.016933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:16.017144 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:16.017245 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:16.017252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.018011 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.018014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:16.018571 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:16.018579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.019035 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.019039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:16.019125 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:16.019713 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:16.019730 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:16.021747 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
W20260810 18:12:16.021991 6267 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:16.016355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:40151 (local address 127.23.65.126:34293)
0810 18:12:16.016526 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:16.016531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:16.016616 (+ 85us) server_negotiation.cc:407] Connection header received
0810 18:12:16.016761 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:16.016764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:16.016814 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:16.016884 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:16.017376 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.017903 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.018693 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.018920 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.019015 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:16.021892 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:40151: BlockingRecv error: recv got EOF from 127.23.65.66:40151 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 18:12:16 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:16 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:16.220172 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:16.208108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:58199)
0810 18:12:16.208394 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:16.208408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:16.208505 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:16.208730 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:16.208732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:16.208935 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:16.209040 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:16.209045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.209849 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.209851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:16.210435 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:16.210444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.210911 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.210914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:16.210981 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:16.217995 (+ 7014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:16.218010 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:16.220003 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":113,"threads_started":1}
W20260810 18:12:16.220283 6267 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:16.208179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:58199 (local address 127.23.65.126:34293)
0810 18:12:16.208307 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:16.208311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:16.208464 (+ 153us) server_negotiation.cc:407] Connection header received
0810 18:12:16.208586 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:16.208588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:16.208636 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:16.208701 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:16.209168 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.209754 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.210548 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.210810 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.210889 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:16.220178 (+ 9289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:58199: BlockingRecv error: recv got EOF from 127.23.65.67:58199 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 18:12:16 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:16 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:16.844643 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:16.839485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:51981)
0810 18:12:16.839799 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:16.839813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:16.839908 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:16.840153 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:16.840156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:16.840342 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:16.840445 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:16.840450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.841131 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.841134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:16.841719 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:16.841728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.842134 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.842138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:16.842211 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:16.842677 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:16.842690 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:16.844487 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":108,"threads_started":1}
W20260810 18:12:16.844748 6272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:16.839633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:51981 (local address 127.23.65.126:34293)
0810 18:12:16.839891 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:16.839895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:16.839908 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:12:16.839988 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:16.839990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:16.840032 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:16.840104 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:16.840533 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.841050 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.841808 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:16.842019 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:16.842084 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:16.844632 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:51981: BlockingRecv error: recv got EOF from 127.23.65.65:51981 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
Aug 10 18:12:17 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:17 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:17.028414 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:17.022751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:45019)
0810 18:12:17.023058 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:17.023072 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:17.023171 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:17.023391 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:17.023393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:17.023609 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:17.023731 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:17.023740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.024513 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.024517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:17.025081 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:17.025090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.025542 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.025546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:17.025645 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:17.026147 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:17.026159 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:17.028197 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":106,"threads_started":1}
W20260810 18:12:17.028473 6272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:17.022890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:45019 (local address 127.23.65.126:34293)
0810 18:12:17.023021 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:17.023025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:17.023135 (+ 110us) server_negotiation.cc:407] Connection header received
0810 18:12:17.023241 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:17.023243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:17.023292 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:17.023357 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:17.023862 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.024395 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.025201 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.025430 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.025525 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:17.028366 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:45019: BlockingRecv error: recv got EOF from 127.23.65.66:45019 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 18:12:17 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:17 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:17.226537 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:17.221134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:55007)
0810 18:12:17.221479 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:17.221494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:17.221652 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:17.221909 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:17.221912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:17.222125 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:17.222229 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:17.222234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.222982 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.222986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:17.223587 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:17.223596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.224028 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.224032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:17.224116 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:17.224624 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:17.224636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:17.226361 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
W20260810 18:12:17.226619 6272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:17.221237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:55007 (local address 127.23.65.126:34293)
0810 18:12:17.221437 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:17.221441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:17.221568 (+ 127us) server_negotiation.cc:407] Connection header received
0810 18:12:17.221750 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:17.221754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:17.221804 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:17.221870 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:17.222349 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.222874 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.223702 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.223924 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.224012 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:17.226514 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:55007: BlockingRecv error: recv got EOF from 127.23.65.67:55007 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 10 18:12:17 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:17 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:17.850618 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:17.845424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:47689)
0810 18:12:17.845714 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:17.845732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:17.845850 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:17.846145 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:17.846148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:17.846338 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:17.846439 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:17.846444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.847166 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.847169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:17.847764 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:17.847774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.848139 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.848141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:17.848197 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:17.848664 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:17.848676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:17.850466 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
W20260810 18:12:17.850721 6277 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:17.845548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:47689 (local address 127.23.65.126:34293)
0810 18:12:17.845918 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:17.845921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:17.845933 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:12:17.845968 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:17.845970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:17.846011 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:17.846083 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:17.846530 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.847065 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.847845 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:17.848057 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:17.848126 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:17.850607 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:47689: BlockingRecv error: recv got EOF from 127.23.65.65:47689 (error 108)
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":107,"threads_started":1}
Aug 10 18:12:18 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:18 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:18.034430 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:18.029218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:54553)
0810 18:12:18.029513 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:18.029527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:18.029639 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:18.029866 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:18.029868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:18.030085 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:18.030220 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:18.030226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.030950 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.030953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:18.031577 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:18.031589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.031985 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.031987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:18.032047 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:18.032543 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:18.032555 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:18.034291 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
W20260810 18:12:18.034535 6277 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:18.029323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:54553 (local address 127.23.65.126:34293)
0810 18:12:18.029487 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:18.029490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:18.029632 (+ 142us) server_negotiation.cc:407] Connection header received
0810 18:12:18.029707 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:18.029709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:18.029765 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:18.029834 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:18.030328 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.030862 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.031679 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.031902 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.031973 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:18.034422 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:54553: BlockingRecv error: recv got EOF from 127.23.65.66:54553 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 18:12:18 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:18 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.67@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:18.232723 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:18.227448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.67:45923)
0810 18:12:18.227757 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:18.227771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:18.227882 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:18.228141 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:18.228143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:18.228362 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:18.228463 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:18.228470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.229240 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.229244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:18.229842 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:18.229851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.230291 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.230295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:18.230389 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:18.230853 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:18.230864 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:18.232593 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":114,"threads_started":1}
W20260810 18:12:18.232863 6277 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:18.227555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:45923 (local address 127.23.65.126:34293)
0810 18:12:18.227712 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:18.227716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:18.227846 (+ 130us) server_negotiation.cc:407] Connection header received
0810 18:12:18.227973 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:18.227976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:18.228031 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:18.228103 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:18.228581 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.229131 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.229960 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.230190 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.230277 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:18.232734 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.67:45923: BlockingRecv error: recv got EOF from 127.23.65.67:45923 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 18:12:18 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:18 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.65@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:18.856529 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:18.851466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.65:59653)
0810 18:12:18.851751 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:18.851764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:18.851844 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:18.852037 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:18.852039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:18.852228 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:18.852327 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:18.852333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.853064 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.853067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:18.853649 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:18.853658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.854026 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.854028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:18.854101 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:18.854575 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:18.854589 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:18.856404 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":106,"threads_started":1}
W20260810 18:12:18.856668 6284 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:18.851552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:59653 (local address 127.23.65.126:34293)
0810 18:12:18.851784 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:18.851788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:18.851815 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:12:18.851911 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:18.851913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:18.851956 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:18.852020 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:18.852413 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.852941 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.853736 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:18.853949 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:18.854014 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:18.856544 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.65:59653: BlockingRecv error: recv got EOF from 127.23.65.65:59653 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":60,"threads_started":1}
Aug 10 18:12:19 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:12:19 dist-test-slave-jpj1 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.65.66@KRBTEST.COM for oryx/127.23.65.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:12:19.040628 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:19.035355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34293 (local address 127.23.65.66:37255)
0810 18:12:19.035689 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:19.035701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:19.035809 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:19.036060 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:19.036063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:19.036264 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:19.036386 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:19.036392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.037154 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.037157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:19.037727 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:19.037737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.038172 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.038175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:19.038258 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:19.038724 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:19.038735 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:19.040456 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.65.126:34293: Server oryx/127.23.65.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":77,"threads_started":1}
W20260810 18:12:19.040700 6284 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:19.035443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:37255 (local address 127.23.65.126:34293)
0810 18:12:19.035614 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:19.035619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:19.035792 (+ 173us) server_negotiation.cc:407] Connection header received
0810 18:12:19.035904 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:19.035907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:19.035954 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:19.036017 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:19.036512 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.037045 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.037849 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.038066 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.038152 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:19.040599 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.65.66:37255: BlockingRecv error: recv got EOF from 127.23.65.66:37255 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:12:19.123507 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 5546
I20260810 18:12:19.137102 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 5787
I20260810 18:12:19.158053 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 6002
I20260810 18:12:19.191329 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 5438
2026-08-10T18:12:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18236 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T18:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:12:19Z Disabled control of system clock
I20260810 18:12:19.240957 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:33027
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:33027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 18:12:19.364629 6299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:19.364907 6299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:19.364939 6299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:19.368947 6299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:12:19.369004 6299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:19.369019 6299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:19.369033 6299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:12:19.369050 6299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:12:19.373793 6299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:33027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:33027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:19.374866 6299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:19.376362 6299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:19.385077 6316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:19.385143 6299 server_base.cc:1034] running on GCE node
W20260810 18:12:19.385087 6313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:19.385098 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:19.385908 6299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:19.387022 6299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:19.388200 6299 hybrid_clock.cc:648] HybridClock initialized: now 1786385539388170 us; error 29 us; skew 500 ppm
I20260810 18:12:19.390041 6299 webserver.cc:460] Webserver started at http://127.23.65.126:45283/ using document root <none> and password file <none>
I20260810 18:12:19.390612 6299 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:19.390676 6299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:19.390887 6299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:19.392992 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "a2bc36cf3c574285bef1c3581cbf51ac"
format_stamp: "Formatted at 2026-08-10 18:12:19 on dist-test-slave-jpj1"
server_key: "bedac77d3bb5b42b09267aa2b1e3e9e2"
server_key_iv: "5d592ba458931cb86fee9b14269b0af1"
server_key_version: "encryptionkey@0"
I20260810 18:12:19.393568 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "a2bc36cf3c574285bef1c3581cbf51ac"
format_stamp: "Formatted at 2026-08-10 18:12:19 on dist-test-slave-jpj1"
server_key: "bedac77d3bb5b42b09267aa2b1e3e9e2"
server_key_iv: "5d592ba458931cb86fee9b14269b0af1"
server_key_version: "encryptionkey@0"
I20260810 18:12:19.397399 6299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:12:19.399839 6325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:19.401054 6299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:12:19.401175 6299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "a2bc36cf3c574285bef1c3581cbf51ac"
format_stamp: "Formatted at 2026-08-10 18:12:19 on dist-test-slave-jpj1"
server_key: "bedac77d3bb5b42b09267aa2b1e3e9e2"
server_key_iv: "5d592ba458931cb86fee9b14269b0af1"
server_key_version: "encryptionkey@0"
I20260810 18:12:19.401314 6299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:19.415138 6299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:19.415864 6299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:19.416038 6299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:19.422996 6299 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:33027
I20260810 18:12:19.423012 6416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:33027 every 8 connection(s)
I20260810 18:12:19.424038 6299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:12:19.427316 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 6299
I20260810 18:12:19.427331 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:12:19.427433 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:12:19.427755 23813 external_mini_cluster.cc:1428] Setting key 94f0ed57119f9e01230c50889bc9c3c8
I20260810 18:12:19.434159 6417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac: Bootstrap starting.
I20260810 18:12:19.436666 6417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac: Neither blocks nor log segments found. Creating new log.
I20260810 18:12:19.437027 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:19.429203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.23.65.126:33027)
0810 18:12:19.429810 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:19.429827 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:19.429864 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:12:19.430561 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:19.430579 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:19.430973 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:19.431255 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:19.431631 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.432616 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.433781 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.434248 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.434424 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:19.435254 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:19.435260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:19.435273 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:12:19.435712 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:19.435770 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:12:19.435791 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:19.435867 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:19.436009 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 18:12:19.436287 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
I20260810 18:12:19.437503 6417 log.cc:824] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac: Log is configured to *not* fsync() on all Append() calls
I20260810 18:12:19.439559 6417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac: No bootstrap required, opened a new log
I20260810 18:12:19.442977 6417 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bc36cf3c574285bef1c3581cbf51ac" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 33027 } }
I20260810 18:12:19.443190 6417 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:12:19.443234 6417 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bc36cf3c574285bef1c3581cbf51ac, State: Initialized, Role: FOLLOWER
I20260810 18:12:19.443678 6417 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bc36cf3c574285bef1c3581cbf51ac" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 33027 } }
I20260810 18:12:19.443823 6417 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:12:19.443892 6417 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:12:19.444022 6417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:12:19.444902 6417 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bc36cf3c574285bef1c3581cbf51ac" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 33027 } }
I20260810 18:12:19.445221 6417 leader_election.cc:302] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2bc36cf3c574285bef1c3581cbf51ac; no voters:
I20260810 18:12:19.445509 6417 leader_election.cc:288] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:12:19.445600 6422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:12:19.445822 6422 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [term 1 LEADER]: Becoming Leader. State: Replica: a2bc36cf3c574285bef1c3581cbf51ac, State: Running, Role: LEADER
I20260810 18:12:19.446147 6422 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bc36cf3c574285bef1c3581cbf51ac" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 33027 } }
I20260810 18:12:19.446713 6417 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:12:19.447741 6426 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2bc36cf3c574285bef1c3581cbf51ac. Latest consensus state: current_term: 1 leader_uuid: "a2bc36cf3c574285bef1c3581cbf51ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bc36cf3c574285bef1c3581cbf51ac" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 33027 } } }
I20260810 18:12:19.447865 6426 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [sys.catalog]: This master's current role is: LEADER
I20260810 18:12:19.448347 6425 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2bc36cf3c574285bef1c3581cbf51ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bc36cf3c574285bef1c3581cbf51ac" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 33027 } } }
I20260810 18:12:19.448541 6425 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac [sys.catalog]: This master's current role is: LEADER
I20260810 18:12:19.448357 6435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:12:19.451390 6435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:12:19.456532 6435 catalog_manager.cc:1347] Generated new cluster ID: 11db0f5c626c4d0088e73906d092b6d1
I20260810 18:12:19.456589 6435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:12:19.475317 6435 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:12:19.476634 6435 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:12:19.482753 6435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2bc36cf3c574285bef1c3581cbf51ac: Generated new TSK 0
I20260810 18:12:19.483568 6435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:12:19.488611 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:33027
--builtin_ntp_servers=127.23.65.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 18:12:19.600883 6461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:19.601131 6461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:19.601187 6461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:19.605149 6461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:19.605202 6461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:19.605274 6461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:12:19.609930 6461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:19.610901 6461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:19.612250 6461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:12:19.621435 6461 server_base.cc:1034] running on GCE node
W20260810 18:12:19.621511 6471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:19.621384 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:19.621384 6470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:19.622036 6461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:19.622681 6461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:19.623839 6461 hybrid_clock.cc:648] HybridClock initialized: now 1786385539623810 us; error 42 us; skew 500 ppm
I20260810 18:12:19.625648 6461 webserver.cc:460] Webserver started at http://127.23.65.65:44623/ using document root <none> and password file <none>
I20260810 18:12:19.626235 6461 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:19.626298 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:19.626525 6461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:19.628634 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "234d4170fb9c4a57827882ab843adca6"
format_stamp: "Formatted at 2026-08-10 18:12:19 on dist-test-slave-jpj1"
server_key: "f45663c1fe8a8d4f88f04bce8353772f"
server_key_iv: "2cc0fdfd36899bb70ebfe7b4e514cfb7"
server_key_version: "encryptionkey@0"
I20260810 18:12:19.629172 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "234d4170fb9c4a57827882ab843adca6"
format_stamp: "Formatted at 2026-08-10 18:12:19 on dist-test-slave-jpj1"
server_key: "f45663c1fe8a8d4f88f04bce8353772f"
server_key_iv: "2cc0fdfd36899bb70ebfe7b4e514cfb7"
server_key_version: "encryptionkey@0"
I20260810 18:12:19.632759 6461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:12:19.635133 6489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:19.636339 6461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:12:19.636446 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "234d4170fb9c4a57827882ab843adca6"
format_stamp: "Formatted at 2026-08-10 18:12:19 on dist-test-slave-jpj1"
server_key: "f45663c1fe8a8d4f88f04bce8353772f"
server_key_iv: "2cc0fdfd36899bb70ebfe7b4e514cfb7"
server_key_version: "encryptionkey@0"
I20260810 18:12:19.636545 6461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:19.664795 6461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:19.665534 6461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:19.665730 6461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:19.666360 6461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:19.667352 6461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:19.667403 6461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:19.667459 6461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:19.667484 6461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:19.676539 6461 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:39615
I20260810 18:12:19.676587 6693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:39615 every 8 connection(s)
I20260810 18:12:19.677444 6461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:12:19.685693 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 6461
I20260810 18:12:19.685808 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:12:19.686115 23813 external_mini_cluster.cc:1428] Setting key de7c49ebd4a0a765a2da61e4a9795d05
I20260810 18:12:19.686482 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:19.679286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:58077 (local address 127.23.65.126:33027)
0810 18:12:19.679447 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:19.679452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:19.680562 (+ 1110us) server_negotiation.cc:407] Connection header received
0810 18:12:19.681922 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:19.681924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:19.681972 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:19.682033 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:19.682912 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.683452 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.684571 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.684782 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.684852 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:19.686027 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:19.686031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:19.686033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:12:19.686046 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:19.686064 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:12:19.686071 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:19.686129 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:19.686274 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 18:12:19.686337 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:12:19.687475 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:19.679734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:33027 (local address 127.23.65.65:58077)
0810 18:12:19.680416 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:19.680458 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:19.681713 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:19.682196 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:19.682203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:19.682287 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:19.682771 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:19.682783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.683559 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.683562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:19.684468 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:19.684478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.684923 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.684927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:19.685110 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:19.685744 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:12:19.685746 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:19.685802 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:12:19.685888 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:19.686139 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:19.686149 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:19.686155 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:12:19.686273 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 18:12:19.686542 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":137,"threads_started":1}
I20260810 18:12:19.688107 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:33027
--builtin_ntp_servers=127.23.65.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 18:12:19.688617 6694 heartbeater.cc:347] Connected to a master server at 127.23.65.126:33027
I20260810 18:12:19.688862 6694 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:19.689450 6694 heartbeater.cc:511] Master 127.23.65.126:33027 requested a full tablet report, sending...
I20260810 18:12:19.691195 6353 ts_manager.cc:194] Registered new tserver with Master: 234d4170fb9c4a57827882ab843adca6 (127.23.65.65:39615)
I20260810 18:12:19.692369 6353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.65:58077
W20260810 18:12:19.802953 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:19.803227 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:19.803264 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:19.807323 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:19.807376 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:19.807432 6698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:12:19.812155 6698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:19.813223 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:19.814627 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:19.824674 6710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:19.824671 6706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:19.824664 6707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:19.825217 6698 server_base.cc:1034] running on GCE node
I20260810 18:12:19.825634 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:19.826288 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:19.827456 6698 hybrid_clock.cc:648] HybridClock initialized: now 1786385539827433 us; error 32 us; skew 500 ppm
I20260810 18:12:19.829432 6698 webserver.cc:460] Webserver started at http://127.23.65.66:33835/ using document root <none> and password file <none>
I20260810 18:12:19.830020 6698 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:19.830093 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:19.830319 6698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:19.832475 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "4c44b966d9aa43c8a5aed402277bb1ba"
format_stamp: "Formatted at 2026-08-10 18:12:19 on dist-test-slave-jpj1"
server_key: "f3317c8427bf9e3b9d967d996be1b1fa"
server_key_iv: "27d42ca8570d6714e2d760284a9a88e6"
server_key_version: "encryptionkey@0"
I20260810 18:12:19.833041 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "4c44b966d9aa43c8a5aed402277bb1ba"
format_stamp: "Formatted at 2026-08-10 18:12:19 on dist-test-slave-jpj1"
server_key: "f3317c8427bf9e3b9d967d996be1b1fa"
server_key_iv: "27d42ca8570d6714e2d760284a9a88e6"
server_key_version: "encryptionkey@0"
I20260810 18:12:19.837044 6698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:12:19.839617 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:19.840895 6698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:12:19.841004 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "4c44b966d9aa43c8a5aed402277bb1ba"
format_stamp: "Formatted at 2026-08-10 18:12:19 on dist-test-slave-jpj1"
server_key: "f3317c8427bf9e3b9d967d996be1b1fa"
server_key_iv: "27d42ca8570d6714e2d760284a9a88e6"
server_key_version: "encryptionkey@0"
I20260810 18:12:19.841102 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:19.886564 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:19.887344 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:19.887566 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:19.888223 6698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:19.889218 6698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:19.889271 6698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:19.889328 6698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:19.889356 6698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:19.897608 6698 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:35839
I20260810 18:12:19.897643 6922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:35839 every 8 connection(s)
I20260810 18:12:19.898511 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:12:19.906229 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 6698
I20260810 18:12:19.906368 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:12:19.906703 23813 external_mini_cluster.cc:1428] Setting key d91b56ae0d95b411b7bc57b341cb9bd0
I20260810 18:12:19.908803 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:19.900399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:52703 (local address 127.23.65.126:33027)
0810 18:12:19.900546 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:19.900550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:19.901690 (+ 1140us) server_negotiation.cc:407] Connection header received
0810 18:12:19.903078 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:19.903081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:19.903131 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:19.903203 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:19.904416 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.905192 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.906709 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.906984 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.907071 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:19.908333 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:19.908337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:19.908340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:12:19.908353 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:19.908376 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:12:19.908386 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:19.908463 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:19.908592 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 18:12:19.908655 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:12:19.909020 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:33027
--builtin_ntp_servers=127.23.65.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 18:12:19.909762 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:19.900835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:33027 (local address 127.23.65.66:52703)
0810 18:12:19.901533 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:19.901575 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:19.902876 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:19.903383 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:19.903393 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:19.903503 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:19.904240 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:19.904262 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.905325 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.905329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:19.906589 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:19.906602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:19.907155 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:19.907159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:19.907354 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:19.908032 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:12:19.908033 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:19.908095 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:12:19.908183 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:19.908463 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:19.908469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:19.908473 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:12:19.908586 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 18:12:19.908863 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260810 18:12:19.910909 6924 heartbeater.cc:347] Connected to a master server at 127.23.65.126:33027
I20260810 18:12:19.911167 6924 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:19.911791 6924 heartbeater.cc:511] Master 127.23.65.126:33027 requested a full tablet report, sending...
I20260810 18:12:19.912874 6353 ts_manager.cc:194] Registered new tserver with Master: 4c44b966d9aa43c8a5aed402277bb1ba (127.23.65.66:35839)
I20260810 18:12:19.913594 6353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.66:52703
W20260810 18:12:20.024124 6927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:20.024405 6927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:20.024442 6927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:20.028637 6927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:20.028693 6927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:20.028748 6927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:12:20.033561 6927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:20.034605 6927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:20.036047 6927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:20.045536 6937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:20.045543 6932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:20.045543 6934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:20.045804 6927 server_base.cc:1034] running on GCE node
I20260810 18:12:20.046329 6927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:20.046959 6927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:20.048113 6927 hybrid_clock.cc:648] HybridClock initialized: now 1786385540048094 us; error 31 us; skew 500 ppm
I20260810 18:12:20.049886 6927 webserver.cc:460] Webserver started at http://127.23.65.67:33363/ using document root <none> and password file <none>
I20260810 18:12:20.050429 6927 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:20.050491 6927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:20.050688 6927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:20.052712 6927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "8378f1d2b7da4de1b74084a1ea684d32"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "563d35c54dfc9e32bf6b42fa2756c62e"
server_key_iv: "586ca5f8079c75a558bc984050cfdbae"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.053251 6927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "8378f1d2b7da4de1b74084a1ea684d32"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "563d35c54dfc9e32bf6b42fa2756c62e"
server_key_iv: "586ca5f8079c75a558bc984050cfdbae"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.056792 6927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:12:20.059171 6945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.060406 6927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:12:20.060518 6927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "8378f1d2b7da4de1b74084a1ea684d32"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "563d35c54dfc9e32bf6b42fa2756c62e"
server_key_iv: "586ca5f8079c75a558bc984050cfdbae"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.060619 6927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:20.078893 6927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:20.079660 6927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:20.079838 6927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:20.080425 6927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:20.081406 6927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:20.081458 6927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.081503 6927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:20.081537 6927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.090154 6927 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:38883
I20260810 18:12:20.090211 7139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:38883 every 8 connection(s)
I20260810 18:12:20.091091 6927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:12:20.096644 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 6927
I20260810 18:12:20.096748 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:12:20.096992 23813 external_mini_cluster.cc:1428] Setting key 7c171fef67d6b418954168d00d7cec04
I20260810 18:12:20.100631 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:20.092966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:44787 (local address 127.23.65.126:33027)
0810 18:12:20.093113 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:20.093117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:20.094165 (+ 1048us) server_negotiation.cc:407] Connection header received
0810 18:12:20.095502 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:20.095504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:20.095585 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:20.095659 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:20.096615 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:20.097176 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:20.098673 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:20.099029 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:20.099142 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:20.100229 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:20.100231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:20.100233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:12:20.100245 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:20.100262 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:12:20.100268 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:20.100339 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:20.100457 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 18:12:20.100512 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:12:20.101532 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:20.093352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:33027 (local address 127.23.65.67:44787)
0810 18:12:20.094017 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:20.094060 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:20.095307 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:20.095832 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:20.095840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:20.095924 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:20.096479 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:20.096499 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:20.097302 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:20.097305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:20.098514 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:20.098527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:20.099187 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:20.099191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:20.099373 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:20.099990 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:12:20.099991 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:20.100035 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:12:20.100104 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:20.100353 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:20.100359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:20.100363 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:12:20.100454 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 18:12:20.100696 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260810 18:12:20.102587 7140 heartbeater.cc:347] Connected to a master server at 127.23.65.126:33027
I20260810 18:12:20.102802 7140 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:20.103304 7140 heartbeater.cc:511] Master 127.23.65.126:33027 requested a full tablet report, sending...
I20260810 18:12:20.104303 6353 ts_manager.cc:194] Registered new tserver with Master: 8378f1d2b7da4de1b74084a1ea684d32 (127.23.65.67:38883)
I20260810 18:12:20.104842 6353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.67:44787
I20260810 18:12:20.110687 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:12:20.113284 6420 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:20.112471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.23.65.126:33027)
0810 18:12:20.112605 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:20.112609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:20.112759 (+ 150us) server_negotiation.cc:407] Connection header received
0810 18:12:20.112861 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:20.112863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:20.112905 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:20.112965 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:20.113160 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38618: BlockingRecv error: recv got EOF from 127.0.0.1:38618 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:12:20.113447 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 6461
I20260810 18:12:20.154058 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 6698
I20260810 18:12:20.164750 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 6927
I20260810 18:12:20.175704 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 6299
2026-08-10T18:12:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (980 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T18:12:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:12:20Z Disabled control of system clock
I20260810 18:12:20.216375 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:37127
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:37127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:12:20.327689 7156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:20.327961 7156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:20.327998 7156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:20.331925 7156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:12:20.331981 7156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:20.331995 7156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:20.332011 7156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:12:20.332031 7156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:12:20.336706 7156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:37127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:20.337777 7156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:20.339344 7156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:20.347952 7171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:20.347952 7168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:20.347956 7167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:20.348028 7156 server_base.cc:1034] running on GCE node
I20260810 18:12:20.348771 7156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:20.349915 7156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:20.351071 7156 hybrid_clock.cc:648] HybridClock initialized: now 1786385540351054 us; error 29 us; skew 500 ppm
I20260810 18:12:20.352905 7156 webserver.cc:460] Webserver started at http://127.23.65.126:37205/ using document root <none> and password file <none>
I20260810 18:12:20.353456 7156 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:20.353524 7156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:20.353718 7156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:20.355827 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "ade04fb2e9e14c70b111e605c2050791"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "dc0620697090289c69c1ce95aa548b7a"
server_key_iv: "eea72bd30245322df75e1d041b2fbd12"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.356374 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "ade04fb2e9e14c70b111e605c2050791"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "dc0620697090289c69c1ce95aa548b7a"
server_key_iv: "eea72bd30245322df75e1d041b2fbd12"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.359935 7156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:12:20.362222 7180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.363375 7156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:12:20.363490 7156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "ade04fb2e9e14c70b111e605c2050791"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "dc0620697090289c69c1ce95aa548b7a"
server_key_iv: "eea72bd30245322df75e1d041b2fbd12"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.363619 7156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:20.367952 7156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:20.368523 7156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:20.368680 7156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:20.375031 7156 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:37127
I20260810 18:12:20.375051 7274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:37127 every 8 connection(s)
I20260810 18:12:20.375952 7156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:12:20.378989 7276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:12:20.381510 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 7156
I20260810 18:12:20.381636 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:12:20.381958 23813 external_mini_cluster.cc:1428] Setting key f62c0a435aba02b643ebe4bf807ea150
I20260810 18:12:20.385094 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791: Bootstrap starting.
I20260810 18:12:20.387414 7276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791: Neither blocks nor log segments found. Creating new log.
I20260810 18:12:20.387578 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:20.383474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.23.65.126:37127)
0810 18:12:20.384174 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:20.384190 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:20.384226 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:12:20.384866 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:20.384883 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:20.385242 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:20.385573 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:20.386329 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:20.386334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:20.386346 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:12:20.386357 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:20.386415 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:12:20.386432 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:20.386561 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:20.386630 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 18:12:20.386863 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":142,"threads_started":1}
I20260810 18:12:20.388334 7276 log.cc:824] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791: Log is configured to *not* fsync() on all Append() calls
I20260810 18:12:20.390288 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791: No bootstrap required, opened a new log
I20260810 18:12:20.393810 7276 raft_consensus.cc:348] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade04fb2e9e14c70b111e605c2050791" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37127 } }
I20260810 18:12:20.394018 7276 raft_consensus.cc:374] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:12:20.394055 7276 raft_consensus.cc:729] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade04fb2e9e14c70b111e605c2050791, State: Initialized, Role: FOLLOWER
I20260810 18:12:20.394426 7276 consensus_queue.cc:260] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade04fb2e9e14c70b111e605c2050791" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37127 } }
I20260810 18:12:20.394549 7276 raft_consensus.cc:388] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:12:20.394593 7276 raft_consensus.cc:482] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:12:20.394661 7276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:12:20.395445 7276 raft_consensus.cc:504] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade04fb2e9e14c70b111e605c2050791" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37127 } }
I20260810 18:12:20.395764 7276 leader_election.cc:302] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ade04fb2e9e14c70b111e605c2050791; no voters:
I20260810 18:12:20.396013 7276 leader_election.cc:288] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:12:20.396124 7283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:12:20.396375 7283 raft_consensus.cc:686] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [term 1 LEADER]: Becoming Leader. State: Replica: ade04fb2e9e14c70b111e605c2050791, State: Running, Role: LEADER
I20260810 18:12:20.396665 7283 consensus_queue.cc:237] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade04fb2e9e14c70b111e605c2050791" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37127 } }
I20260810 18:12:20.397178 7276 sys_catalog.cc:564] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:12:20.398234 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ade04fb2e9e14c70b111e605c2050791. Latest consensus state: current_term: 1 leader_uuid: "ade04fb2e9e14c70b111e605c2050791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade04fb2e9e14c70b111e605c2050791" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37127 } } }
I20260810 18:12:20.398361 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [sys.catalog]: This master's current role is: LEADER
I20260810 18:12:20.398630 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ade04fb2e9e14c70b111e605c2050791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade04fb2e9e14c70b111e605c2050791" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37127 } } }
I20260810 18:12:20.398723 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791 [sys.catalog]: This master's current role is: LEADER
I20260810 18:12:20.398856 7298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:12:20.401969 7298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:12:20.407244 7298 catalog_manager.cc:1347] Generated new cluster ID: a53020e6dec84b12b905427e39abbcd2
I20260810 18:12:20.407308 7298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:12:20.415809 7298 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:12:20.416147 7298 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:12:20.423722 7298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ade04fb2e9e14c70b111e605c2050791: Generated new TSK 0
I20260810 18:12:20.424437 7298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:12:20.430212 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37127
--builtin_ntp_servers=127.23.65.84:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:12:20.542788 7314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:20.543078 7314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:20.543121 7314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:20.547158 7314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:20.547209 7314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:20.547273 7314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:12:20.552045 7314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:20.553099 7314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:20.554477 7314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:20.563714 7320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:20.563723 7323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:20.563759 7314 server_base.cc:1034] running on GCE node
W20260810 18:12:20.563723 7321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:20.564414 7314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:20.565006 7314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:20.566201 7314 hybrid_clock.cc:648] HybridClock initialized: now 1786385540566177 us; error 32 us; skew 500 ppm
I20260810 18:12:20.568022 7314 webserver.cc:460] Webserver started at http://127.23.65.65:40437/ using document root <none> and password file <none>
I20260810 18:12:20.568564 7314 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:20.568627 7314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:20.568841 7314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:20.570845 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "bad12f54167a4d669a4efa5fc7573010"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "c9c2253527a2d14a47907719cb370741"
server_key_iv: "0988fef2d0b7f83c9367bce9f87cefeb"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.571380 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "bad12f54167a4d669a4efa5fc7573010"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "c9c2253527a2d14a47907719cb370741"
server_key_iv: "0988fef2d0b7f83c9367bce9f87cefeb"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.574922 7314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:12:20.577379 7336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.578540 7314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:12:20.578644 7314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "bad12f54167a4d669a4efa5fc7573010"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "c9c2253527a2d14a47907719cb370741"
server_key_iv: "0988fef2d0b7f83c9367bce9f87cefeb"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.578744 7314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:20.583249 7314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:20.583899 7314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:20.584056 7314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:20.584604 7314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:20.585523 7314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:20.585573 7314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.585616 7314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:20.585645 7314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.594076 7314 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:40203
I20260810 18:12:20.594110 7520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:40203 every 8 connection(s)
I20260810 18:12:20.594952 7314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:12:20.596936 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 7314
I20260810 18:12:20.597038 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:12:20.597316 23813 external_mini_cluster.cc:1428] Setting key e3e80f1f0d88fb606dba5d33e11d2d6b
I20260810 18:12:20.599255 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37127
--builtin_ntp_servers=127.23.65.84:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 18:12:20.601239 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:20.596682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:54993 (local address 127.23.65.126:37127)
0810 18:12:20.596851 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:20.596856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:20.598038 (+ 1182us) server_negotiation.cc:407] Connection header received
0810 18:12:20.599484 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:20.599487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:20.599524 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:20.599621 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:20.600818 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:20.600822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:20.600824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:12:20.600838 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:20.600862 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:12:20.600873 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:20.600966 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:20.601101 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 18:12:20.601135 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:12:20.602335 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:20.597120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:37127 (local address 127.23.65.65:54993)
0810 18:12:20.597882 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:20.597936 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:20.599254 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:20.599927 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:20.599933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:20.600019 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:20.600557 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:12:20.600558 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:20.600598 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:12:20.600667 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:20.600964 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:20.600973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:20.600979 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:12:20.601147 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 18:12:20.601399 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":121,"threads_started":1}
I20260810 18:12:20.603554 7521 heartbeater.cc:347] Connected to a master server at 127.23.65.126:37127
I20260810 18:12:20.603776 7521 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:20.604164 7521 heartbeater.cc:511] Master 127.23.65.126:37127 requested a full tablet report, sending...
I20260810 18:12:20.605626 7212 ts_manager.cc:194] Registered new tserver with Master: bad12f54167a4d669a4efa5fc7573010 (127.23.65.65:40203)
W20260810 18:12:20.710767 7525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:20.711021 7525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:20.711076 7525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:20.715027 7525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:20.715075 7525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:20.715148 7525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:12:20.719763 7525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:20.720790 7525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:20.722146 7525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:20.731395 7536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:20.731465 7525 server_base.cc:1034] running on GCE node
W20260810 18:12:20.731382 7538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:20.731382 7543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:20.732038 7525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:20.732690 7525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:20.733867 7525 hybrid_clock.cc:648] HybridClock initialized: now 1786385540733844 us; error 31 us; skew 500 ppm
I20260810 18:12:20.735854 7525 webserver.cc:460] Webserver started at http://127.23.65.66:37365/ using document root <none> and password file <none>
I20260810 18:12:20.736440 7525 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:20.736507 7525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:20.736726 7525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:20.738874 7525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "40f8b3fb57c04147a9e3e46aa32ef8e5"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "1e306a0ae87322185992a5933194c29b"
server_key_iv: "79fac55f4979e3e824862035babe3bd3"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.739413 7525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "40f8b3fb57c04147a9e3e46aa32ef8e5"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "1e306a0ae87322185992a5933194c29b"
server_key_iv: "79fac55f4979e3e824862035babe3bd3"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.743086 7525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:12:20.745534 7553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.746697 7525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:12:20.746798 7525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "40f8b3fb57c04147a9e3e46aa32ef8e5"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "1e306a0ae87322185992a5933194c29b"
server_key_iv: "79fac55f4979e3e824862035babe3bd3"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.746912 7525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:20.751340 7525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:20.751950 7525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:20.752103 7525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:20.752615 7525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:20.753505 7525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:20.753556 7525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.753605 7525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:20.753633 7525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.761852 7525 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:38493
I20260810 18:12:20.761904 7752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:38493 every 8 connection(s)
I20260810 18:12:20.762756 7525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:12:20.766078 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 7525
I20260810 18:12:20.766191 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:12:20.766458 23813 external_mini_cluster.cc:1428] Setting key 341a4020c259083273b88fb91bbee8b1
I20260810 18:12:20.768472 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37127
--builtin_ntp_servers=127.23.65.84:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 18:12:20.769169 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:20.764435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:36501 (local address 127.23.65.126:37127)
0810 18:12:20.764598 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:20.764603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:20.765592 (+ 989us) server_negotiation.cc:407] Connection header received
0810 18:12:20.766984 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:20.766988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:20.767034 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:20.767122 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:20.768748 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:20.768751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:20.768754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:12:20.768766 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:20.768790 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:12:20.768799 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:20.768877 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:20.768994 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 18:12:20.769035 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:12:20.770012 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:20.764820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:37127 (local address 127.23.65.66:36501)
0810 18:12:20.765458 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:20.765498 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:20.766744 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:20.767803 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:20.767810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:20.767900 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:20.768475 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:12:20.768477 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:20.768518 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:12:20.768593 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:20.768878 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:20.768883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:20.768887 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:12:20.769012 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 18:12:20.769201 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
I20260810 18:12:20.771108 7753 heartbeater.cc:347] Connected to a master server at 127.23.65.126:37127
I20260810 18:12:20.771327 7753 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:20.771749 7753 heartbeater.cc:511] Master 127.23.65.126:37127 requested a full tablet report, sending...
I20260810 18:12:20.772820 7212 ts_manager.cc:194] Registered new tserver with Master: 40f8b3fb57c04147a9e3e46aa32ef8e5 (127.23.65.66:38493)
W20260810 18:12:20.881564 7757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:20.881879 7757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:20.881937 7757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:20.885984 7757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:20.886039 7757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:20.886114 7757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:12:20.890820 7757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:20.891927 7757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:20.893345 7757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:20.903071 7771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:20.903066 7768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:20.903071 7769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:20.903218 7757 server_base.cc:1034] running on GCE node
I20260810 18:12:20.903764 7757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:20.904368 7757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:20.905522 7757 hybrid_clock.cc:648] HybridClock initialized: now 1786385540905502 us; error 31 us; skew 500 ppm
I20260810 18:12:20.907433 7757 webserver.cc:460] Webserver started at http://127.23.65.67:46509/ using document root <none> and password file <none>
I20260810 18:12:20.908118 7757 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:20.908210 7757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:20.908486 7757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:20.910858 7757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "1858e466d5b94ca996eebb113d5ce444"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "88b0781cbe7e80248c92b5a93927aba0"
server_key_iv: "555f915b062c93ba35e5dd031b25ed6d"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.911458 7757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "1858e466d5b94ca996eebb113d5ce444"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "88b0781cbe7e80248c92b5a93927aba0"
server_key_iv: "555f915b062c93ba35e5dd031b25ed6d"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.915004 7757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:12:20.917552 7780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.918715 7757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:12:20.918820 7757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "1858e466d5b94ca996eebb113d5ce444"
format_stamp: "Formatted at 2026-08-10 18:12:20 on dist-test-slave-jpj1"
server_key: "88b0781cbe7e80248c92b5a93927aba0"
server_key_iv: "555f915b062c93ba35e5dd031b25ed6d"
server_key_version: "encryptionkey@0"
I20260810 18:12:20.918917 7757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:20.923354 7757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:20.923990 7757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:20.924149 7757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:20.924670 7757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:20.925570 7757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:20.925619 7757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.925669 7757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:20.925697 7757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:20.934051 7757 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:36955
I20260810 18:12:20.934072 7983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:36955 every 8 connection(s)
I20260810 18:12:20.934950 7757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:12:20.936571 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 7757
I20260810 18:12:20.936666 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:12:20.936980 23813 external_mini_cluster.cc:1428] Setting key a29a52369454aa0ea6b89f83130d818a
I20260810 18:12:20.941001 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:20.936782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:38033 (local address 127.23.65.126:37127)
0810 18:12:20.936965 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:20.936970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:20.938154 (+ 1184us) server_negotiation.cc:407] Connection header received
0810 18:12:20.939504 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:20.939506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:20.939604 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:20.939676 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:20.940649 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:20.940653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:20.940655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:12:20.940665 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:20.940685 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:12:20.940693 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:20.940757 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:20.940862 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 18:12:20.940896 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:12:20.941838 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:20.937237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:37127 (local address 127.23.65.67:38033)
0810 18:12:20.937995 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:20.938050 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:20.939317 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:20.939838 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:20.939845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:20.939934 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:20.940415 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:12:20.940417 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:20.940456 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:12:20.940525 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:20.940754 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:20.940758 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:20.940762 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:12:20.940895 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 18:12:20.941092 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":132,"threads_started":1}
I20260810 18:12:20.942833 7985 heartbeater.cc:347] Connected to a master server at 127.23.65.126:37127
I20260810 18:12:20.943039 7985 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:20.943420 7985 heartbeater.cc:511] Master 127.23.65.126:37127 requested a full tablet report, sending...
I20260810 18:12:20.944368 7212 ts_manager.cc:194] Registered new tserver with Master: 1858e466d5b94ca996eebb113d5ce444 (127.23.65.67:36955)
I20260810 18:12:20.950369 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:12:20.953135 7279 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:12:20.952259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.23.65.126:37127)
0810 18:12:20.952383 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:20.952387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:20.952543 (+ 156us) server_negotiation.cc:407] Connection header received
0810 18:12:20.952655 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:20.952658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:20.952693 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:20.952758 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:20.953017 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35794: BlockingRecv error: recv got EOF from 127.0.0.1:35794 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:12:20.953395 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 7314
I20260810 18:12:20.980438 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 7525
I20260810 18:12:20.994535 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 7757
I20260810 18:12:21.005481 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 7156
2026-08-10T18:12:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (826 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:12:21 dist-test-slave-jpj1 krb5kdc[8003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:12:21 dist-test-slave-jpj1 krb5kdc[8003](info): set up 2 sockets
Aug 10 18:12:21 dist-test-slave-jpj1 krb5kdc[8003](info): commencing operation
krb5kdc: starting...
W20260810 18:12:22.168366 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:12:22 dist-test-slave-jpj1 krb5kdc[8003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:12:22.292728 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:45551
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:45551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:22.406819 8034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:22.407092 8034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:22.407147 8034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:22.411132 8034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:12:22.411196 8034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:22.411223 8034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:22.411243 8034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:12:22.411264 8034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:12:22.416534 8034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:45551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:45551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:22.417591 8034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:22.418634 8034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:22.427920 8043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:22.427914 8042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:22.427917 8048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:22.428272 8034 server_base.cc:1034] running on GCE node
I20260810 18:12:22.428943 8034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:22.430140 8034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:22.431335 8034 hybrid_clock.cc:648] HybridClock initialized: now 1786385542431298 us; error 46 us; skew 500 ppm
Aug 10 18:12:22 dist-test-slave-jpj1 krb5kdc[8003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385542, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:22.438593 8034 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:12:22.439575 8034 webserver.cc:460] Webserver started at http://127.23.65.126:39527/ using document root <none> and password file <none>
I20260810 18:12:22.440143 8034 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:22.440207 8034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:22.440382 8034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:22.442442 8034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "72a568282b7346c6b550ce567eb9f59d"
format_stamp: "Formatted at 2026-08-10 18:12:22 on dist-test-slave-jpj1"
server_key: "9a1a14d544871ea38c5be8c7b61b0a9f"
server_key_iv: "30b6c1c123e13e3860b1e7fe93762234"
server_key_version: "encryptionkey@0"
I20260810 18:12:22.442979 8034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "72a568282b7346c6b550ce567eb9f59d"
format_stamp: "Formatted at 2026-08-10 18:12:22 on dist-test-slave-jpj1"
server_key: "9a1a14d544871ea38c5be8c7b61b0a9f"
server_key_iv: "30b6c1c123e13e3860b1e7fe93762234"
server_key_version: "encryptionkey@0"
I20260810 18:12:22.446787 8034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:12:22.449250 8056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:22.450517 8034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:12:22.450629 8034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "72a568282b7346c6b550ce567eb9f59d"
format_stamp: "Formatted at 2026-08-10 18:12:22 on dist-test-slave-jpj1"
server_key: "9a1a14d544871ea38c5be8c7b61b0a9f"
server_key_iv: "30b6c1c123e13e3860b1e7fe93762234"
server_key_version: "encryptionkey@0"
I20260810 18:12:22.450721 8034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:22.473320 8034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:22.476563 8034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:22.476724 8034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:22.483743 8034 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:45551
I20260810 18:12:22.483777 8143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:45551 every 8 connection(s)
I20260810 18:12:22.484709 8034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:12:22.487525 8148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:12:22.489146 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 8034
I20260810 18:12:22.489256 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:12:22.489533 23813 external_mini_cluster.cc:1428] Setting key b0303eff6ead3489a671c2ed9c3120b5
I20260810 18:12:22.494076 8148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d: Bootstrap starting.
I20260810 18:12:22.496635 8148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d: Neither blocks nor log segments found. Creating new log.
I20260810 18:12:22.497398 8148 log.cc:824] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:12:22.499754 8148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d: No bootstrap required, opened a new log
Aug 10 18:12:22 dist-test-slave-jpj1 krb5kdc[8003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:12:22.503834 8148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a568282b7346c6b550ce567eb9f59d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45551 } }
I20260810 18:12:22.504127 8148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:12:22.504205 8148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a568282b7346c6b550ce567eb9f59d, State: Initialized, Role: FOLLOWER
I20260810 18:12:22.504678 8148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a568282b7346c6b550ce567eb9f59d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45551 } }
I20260810 18:12:22.504812 8148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:12:22.504892 8148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:12:22.504994 8148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:12:22.506201 8148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a568282b7346c6b550ce567eb9f59d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45551 } }
I20260810 18:12:22.506594 8148 leader_election.cc:302] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72a568282b7346c6b550ce567eb9f59d; no voters:
I20260810 18:12:22.506980 8148 leader_election.cc:288] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:12:22.507071 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:22.490893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.23.65.126:45551)
0810 18:12:22.491306 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:22.491325 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:22.491369 (+ 44us) server_negotiation.cc:407] Connection header received
0810 18:12:22.492206 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:22.492230 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:22.492640 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:22.493010 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:22.493638 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:22.495073 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:22.495965 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:22.496638 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:22.496813 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:22.500622 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:22.500629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:22.500674 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:22.500709 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:22.503628 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:22.504203 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:22.504210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:22.504217 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:22.504382 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:22.504716 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:22.504720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:22.504722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:22.505323 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:22.505538 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:22.505837 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 18:12:22.506132 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":97,"threads_started":1}
I20260810 18:12:22.508390 8148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:12:22.507078 8156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:12:22.508774 8156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [term 1 LEADER]: Becoming Leader. State: Replica: 72a568282b7346c6b550ce567eb9f59d, State: Running, Role: LEADER
I20260810 18:12:22.509122 8156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a568282b7346c6b550ce567eb9f59d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45551 } }
I20260810 18:12:22.510854 8159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72a568282b7346c6b550ce567eb9f59d. Latest consensus state: current_term: 1 leader_uuid: "72a568282b7346c6b550ce567eb9f59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a568282b7346c6b550ce567eb9f59d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45551 } } }
I20260810 18:12:22.511140 8159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [sys.catalog]: This master's current role is: LEADER
I20260810 18:12:22.511471 8158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72a568282b7346c6b550ce567eb9f59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a568282b7346c6b550ce567eb9f59d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 45551 } } }
W20260810 18:12:22.514022 8183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 18:12:22.514070 8158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d [sys.catalog]: This master's current role is: LEADER
W20260810 18:12:22.514132 8183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:12:22.514273 8188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:12:22.515156 8188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:12:22.522390 8188 catalog_manager.cc:1347] Generated new cluster ID: b6a5704ae4754bbfbfb1529ceea7c4f7
I20260810 18:12:22.522492 8188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:12:22.566606 8188 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:12:22.567663 8188 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:12:22.579890 8188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d: Generated new TSK 0
I20260810 18:12:22.580691 8188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:12:22.611922 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45551
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:22.725302 8203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:22.725561 8203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:22.725603 8203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:22.729615 8203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:22.729678 8203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:22.729741 8203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:12:22.734833 8203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:22.735877 8203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:22.736863 8203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:22.746861 8215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:22.746826 8212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:22.746935 8203 server_base.cc:1034] running on GCE node
W20260810 18:12:22.746842 8213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:22.747479 8203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:22.748198 8203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:22.749354 8203 hybrid_clock.cc:648] HybridClock initialized: now 1786385542749332 us; error 37 us; skew 500 ppm
Aug 10 18:12:22 dist-test-slave-jpj1 krb5kdc[8003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385542, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:22.752066 8203 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:12:22.752918 8203 webserver.cc:460] Webserver started at http://127.23.65.65:46053/ using document root <none> and password file <none>
I20260810 18:12:22.753453 8203 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:22.753515 8203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:22.753731 8203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:22.755865 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "5a5e67145ee04bdd90ec14c02a0482e8"
format_stamp: "Formatted at 2026-08-10 18:12:22 on dist-test-slave-jpj1"
server_key: "53ba5047603e971645901e20f3c9aeb2"
server_key_iv: "04a71921af7d470ebaf2484ddbf438c9"
server_key_version: "encryptionkey@0"
I20260810 18:12:22.756404 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "5a5e67145ee04bdd90ec14c02a0482e8"
format_stamp: "Formatted at 2026-08-10 18:12:22 on dist-test-slave-jpj1"
server_key: "53ba5047603e971645901e20f3c9aeb2"
server_key_iv: "04a71921af7d470ebaf2484ddbf438c9"
server_key_version: "encryptionkey@0"
I20260810 18:12:22.759980 8203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:12:22.762231 8235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:22.763398 8203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:12:22.763499 8203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "5a5e67145ee04bdd90ec14c02a0482e8"
format_stamp: "Formatted at 2026-08-10 18:12:22 on dist-test-slave-jpj1"
server_key: "53ba5047603e971645901e20f3c9aeb2"
server_key_iv: "04a71921af7d470ebaf2484ddbf438c9"
server_key_version: "encryptionkey@0"
I20260810 18:12:22.763621 8203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:22.779496 8203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:22.782428 8203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:22.782582 8203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:22.783116 8203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:22.784019 8203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:22.784068 8203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:22.784108 8203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:22.784129 8203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:22.793085 8203 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:46483
I20260810 18:12:22.793143 8419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:46483 every 8 connection(s)
I20260810 18:12:22.793968 8203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:12:22.798421 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 8203
I20260810 18:12:22.798543 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:12:22.798864 23813 external_mini_cluster.cc:1428] Setting key 79907a6d4a14bd3c6fba340ad9e38498
Aug 10 18:12:22 dist-test-slave-jpj1 krb5kdc[8003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385542, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:12:22.808177 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:22.795829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:50141 (local address 127.23.65.126:45551)
0810 18:12:22.795963 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:22.795967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:22.796780 (+ 813us) server_negotiation.cc:407] Connection header received
0810 18:12:22.798142 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:22.798146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:22.798205 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:22.798278 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:22.799615 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:22.800289 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:22.801801 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:22.802032 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:22.802109 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:22.804657 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:22.804662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:22.804665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:22.804694 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:22.806209 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:22.806671 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:22.806675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:22.806677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:22.806771 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:22.807050 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:22.807053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:22.807055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:22.807282 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:22.807378 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:22.807880 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 18:12:22.808010 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:12:22.808959 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:22.796185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:45551 (local address 127.23.65.65:50141)
0810 18:12:22.796629 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:22.796677 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:22.797935 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:22.798443 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:22.798450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:22.798860 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:22.799427 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:22.799449 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:22.800422 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:22.800425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:22.801672 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:22.801685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:22.802174 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:22.802179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:22.802380 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:22.802944 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:22.802960 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:22.804496 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:22.806323 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:22.806328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:22.806339 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:22.806581 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:22.806849 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:22.806851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:22.806852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:22.806971 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:22.807377 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:22.807382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:22.807644 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 18:12:22.807807 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 18:12:22.808086 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:12:22.810210 8421 heartbeater.cc:347] Connected to a master server at 127.23.65.126:45551
I20260810 18:12:22.810437 8421 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:22.810909 8421 heartbeater.cc:511] Master 127.23.65.126:45551 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
I20260810 18:12:22.812508 8084 ts_manager.cc:194] Registered new tserver with Master: 5a5e67145ee04bdd90ec14c02a0482e8 (127.23.65.65:46483)
I20260810 18:12:22.813623 8084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:50141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:12:22.828354 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45551
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:22.943337 8427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:22.943681 8427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:22.943753 8427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:22.947841 8427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:22.947904 8427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:22.947990 8427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:12:22.953331 8427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:22.954397 8427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:22.955420 8427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:22.965248 8435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:22.965307 8434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:22.965276 8441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:22.965583 8427 server_base.cc:1034] running on GCE node
I20260810 18:12:22.966001 8427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:22.966607 8427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:22.967764 8427 hybrid_clock.cc:648] HybridClock initialized: now 1786385542967753 us; error 34 us; skew 500 ppm
Aug 10 18:12:22 dist-test-slave-jpj1 krb5kdc[8003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385542, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:22.970376 8427 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:12:22.971213 8427 webserver.cc:460] Webserver started at http://127.23.65.66:44815/ using document root <none> and password file <none>
I20260810 18:12:22.971783 8427 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:22.971861 8427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:22.972067 8427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:22.974028 8427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "031b7b8c983d469486f3230632cc3d3a"
format_stamp: "Formatted at 2026-08-10 18:12:22 on dist-test-slave-jpj1"
server_key: "8e777d802c688d447cf7413b0644e7c6"
server_key_iv: "853099edfcdc940497afe5ee507233cd"
server_key_version: "encryptionkey@0"
I20260810 18:12:22.974568 8427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "031b7b8c983d469486f3230632cc3d3a"
format_stamp: "Formatted at 2026-08-10 18:12:22 on dist-test-slave-jpj1"
server_key: "8e777d802c688d447cf7413b0644e7c6"
server_key_iv: "853099edfcdc940497afe5ee507233cd"
server_key_version: "encryptionkey@0"
I20260810 18:12:22.978197 8427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 18:12:22.980664 8457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:22.981943 8427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:12:22.982071 8427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "031b7b8c983d469486f3230632cc3d3a"
format_stamp: "Formatted at 2026-08-10 18:12:22 on dist-test-slave-jpj1"
server_key: "8e777d802c688d447cf7413b0644e7c6"
server_key_iv: "853099edfcdc940497afe5ee507233cd"
server_key_version: "encryptionkey@0"
I20260810 18:12:22.982178 8427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:22.996747 8427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:22.999778 8427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:22.999958 8427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:23.000516 8427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:23.001339 8427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:23.001389 8427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:23.001442 8427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:23.001467 8427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:23.010210 8427 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:44327
I20260810 18:12:23.010280 8667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:44327 every 8 connection(s)
I20260810 18:12:23.011154 8427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:12:23.015326 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 8427
I20260810 18:12:23.015446 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:12:23.015803 23813 external_mini_cluster.cc:1428] Setting key a45d57aa0642a76e56dd6b112c6ecdec
Aug 10 18:12:23 dist-test-slave-jpj1 krb5kdc[8003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385542, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:12:23.025508 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.012939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:58755 (local address 127.23.65.126:45551)
0810 18:12:23.013073 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:23.013077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:23.013897 (+ 820us) server_negotiation.cc:407] Connection header received
0810 18:12:23.015422 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:23.015426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:23.015488 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:23.015608 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:23.017001 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.017616 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.018728 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.018952 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.019042 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.021668 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:23.021671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:23.021673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:23.021702 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:23.023106 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:23.023673 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:23.023678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:23.023680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:23.023783 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:23.024126 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:23.024130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:23.024131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:23.024455 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:23.024566 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:23.025171 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 18:12:23.025331 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:12:23.026409 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.013300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:45551 (local address 127.23.65.66:58755)
0810 18:12:23.013740 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:23.013800 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:23.015183 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:23.015798 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:23.015807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:23.016294 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:23.016862 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.016877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.017741 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.017744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.018607 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.018617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.019095 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.019099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.019260 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.019873 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:23.019890 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:23.021466 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:23.023227 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:23.023232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:23.023246 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:23.023529 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:23.023886 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:23.023888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:23.023890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:23.024041 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:23.024600 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:23.024608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:23.024919 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 18:12:23.025093 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 18:12:23.025420 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:12:23.027740 8668 heartbeater.cc:347] Connected to a master server at 127.23.65.126:45551
I20260810 18:12:23.027994 8668 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:23.028491 8668 heartbeater.cc:511] Master 127.23.65.126:45551 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
I20260810 18:12:23.029626 8084 ts_manager.cc:194] Registered new tserver with Master: 031b7b8c983d469486f3230632cc3d3a (127.23.65.66:44327)
I20260810 18:12:23.030284 8084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:58755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:12:23.044673 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45551
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:12:23.157331 8682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:12:23.157593 8682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:12:23.157634 8682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:12:23.161608 8682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:12:23.161666 8682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:12:23.161729 8682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:12:23.166978 8682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:45551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:12:23.168071 8682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:12:23.169080 8682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:12:23.178606 8695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:23.178651 8682 server_base.cc:1034] running on GCE node
W20260810 18:12:23.178610 8691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:12:23.178610 8690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:12:23.179324 8682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:12:23.180079 8682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:12:23.181248 8682 hybrid_clock.cc:648] HybridClock initialized: now 1786385543181226 us; error 38 us; skew 500 ppm
Aug 10 18:12:23 dist-test-slave-jpj1 krb5kdc[8003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385543, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:12:23.184207 8682 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:12:23.185155 8682 webserver.cc:460] Webserver started at http://127.23.65.67:33197/ using document root <none> and password file <none>
I20260810 18:12:23.185721 8682 fs_manager.cc:359] Metadata directory not provided
I20260810 18:12:23.185791 8682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:12:23.186035 8682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:12:23.188405 8682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "70ea3a6b2fda40e88848a2ecf673913a"
format_stamp: "Formatted at 2026-08-10 18:12:23 on dist-test-slave-jpj1"
server_key: "12987ad192204c2cdf86b153e740f5b9"
server_key_iv: "ed0a8f8585bf405eb0e2832367e0f448"
server_key_version: "encryptionkey@0"
I20260810 18:12:23.189008 8682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "70ea3a6b2fda40e88848a2ecf673913a"
format_stamp: "Formatted at 2026-08-10 18:12:23 on dist-test-slave-jpj1"
server_key: "12987ad192204c2cdf86b153e740f5b9"
server_key_iv: "ed0a8f8585bf405eb0e2832367e0f448"
server_key_version: "encryptionkey@0"
I20260810 18:12:23.192847 8682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:12:23.195220 8706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:23.196480 8682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:12:23.196607 8682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "70ea3a6b2fda40e88848a2ecf673913a"
format_stamp: "Formatted at 2026-08-10 18:12:23 on dist-test-slave-jpj1"
server_key: "12987ad192204c2cdf86b153e740f5b9"
server_key_iv: "ed0a8f8585bf405eb0e2832367e0f448"
server_key_version: "encryptionkey@0"
I20260810 18:12:23.196705 8682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:12:23.219300 8682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:12:23.222389 8682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:12:23.222572 8682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:12:23.223227 8682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:12:23.224131 8682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:12:23.224180 8682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:23.224236 8682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:12:23.224259 8682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:12:23.233343 8682 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:39785
I20260810 18:12:23.233378 8894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:39785 every 8 connection(s)
I20260810 18:12:23.234292 8682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:12:23.241074 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 8682
I20260810 18:12:23.241200 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:12:23.241467 23813 external_mini_cluster.cc:1428] Setting key 38b250fbb80a6606f5ac9b79cd6adf93
Aug 10 18:12:23 dist-test-slave-jpj1 krb5kdc[8003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385543, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:12:23.247742 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.236206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:53061 (local address 127.23.65.126:45551)
0810 18:12:23.236364 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:23.236368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:23.237227 (+ 859us) server_negotiation.cc:407] Connection header received
0810 18:12:23.238597 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:23.238600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:23.238649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:23.238723 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:23.239909 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.240459 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.241635 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.241865 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.241956 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.244501 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:23.244505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:23.244506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:23.244531 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:23.245808 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:23.246278 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:23.246282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:23.246283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:23.246359 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:23.246668 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:23.246672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:23.246674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:23.246865 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:23.246952 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:23.247432 (+ 480us) server_negotiation.cc:299] Negotiation successful
0810 18:12:23.247580 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:12:23.248543 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.236571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:45551 (local address 127.23.65.67:53061)
0810 18:12:23.237078 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:23.237121 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:23.238393 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:23.238883 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:23.238889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:23.239247 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:12:23.239774 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.239790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.240567 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.240569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.241506 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.241518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.242006 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.242010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.242172 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.242764 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:12:23.242797 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:12:23.244352 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:12:23.245918 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:23.245924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:23.245939 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:23.246187 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:23.246440 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:12:23.246443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:12:23.246445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:12:23.246583 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:12:23.246965 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:12:23.246972 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:12:23.247203 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 18:12:23.247373 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 18:12:23.247706 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":129,"threads_started":1}
I20260810 18:12:23.249548 8897 heartbeater.cc:347] Connected to a master server at 127.23.65.126:45551
I20260810 18:12:23.249759 8897 heartbeater.cc:464] Registering TS with master...
I20260810 18:12:23.250195 8897 heartbeater.cc:511] Master 127.23.65.126:45551 requested a full tablet report, sending...
I20260810 18:12:23.251164 8084 ts_manager.cc:194] Registered new tserver with Master: 70ea3a6b2fda40e88848a2ecf673913a (127.23.65.67:39785)
I20260810 18:12:23.251727 8084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:53061
I20260810 18:12:23.255299 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:12:23.264153 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.257273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.23.65.126:45551)
0810 18:12:23.257409 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:23.257414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:23.257599 (+ 185us) server_negotiation.cc:407] Connection header received
0810 18:12:23.257709 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:23.257712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:23.257754 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:23.257821 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:12:23.258530 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.259034 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.259898 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.260111 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.260193 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.261455 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:12:23.261458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:12:23.261461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:12:23.261486 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:12:23.262645 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:23.263032 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:23.263034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:23.263035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:23.263107 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:12:23.263310 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:12:23.263312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:12:23.263313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:12:23.263511 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:12:23.263661 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:23.263874 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 18:12:23.263990 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:12:23.266824 8084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:12:23.269933 8084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:12:23.284389 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.278801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.23.65.65:46483)
0810 18:12:23.279181 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:23.279197 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:23.279219 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:12:23.279292 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:23.279297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:23.279466 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:23.279886 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:12:23.280041 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.281173 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.283095 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.283711 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.283813 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.283970 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:23.284141 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 18:12:23.284246 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":65,"threads_started":1}
I20260810 18:12:23.284386 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.277891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:39785 (local address 127.0.0.1:58770)
0810 18:12:23.278235 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:23.278293 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:23.278431 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:23.279420 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:23.279426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:23.279449 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:12:23.279657 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.279664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.281342 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.281344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.282867 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.282877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.283986 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.283990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.284076 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.284098 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:12:23.284148 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 18:12:23.284228 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":313,"thread_start_us":76,"threads_started":1}
I20260810 18:12:23.284385 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.277970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:46483 (local address 127.0.0.1:44362)
0810 18:12:23.278841 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:23.278855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:23.278938 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:23.279644 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:23.279647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:23.279665 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:12:23.279764 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.279771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.281303 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.281306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.282951 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.282958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.283826 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.283829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.283949 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.283987 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 18:12:23.284088 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 18:12:23.284214 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":92,"thread_start_us":84,"threads_started":1}
I20260810 18:12:23.284725 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.279029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.23.65.66:44327)
0810 18:12:23.279363 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:23.279376 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:23.279731 (+ 355us) server_negotiation.cc:407] Connection header received
0810 18:12:23.279951 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:23.279955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:23.280086 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:23.280215 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:12:23.280417 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.281122 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.283072 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.283863 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.283952 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.284095 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:23.284477 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 18:12:23.284578 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":121,"threads_started":1}
I20260810 18:12:23.285014 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.278257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.23.65.67:39785)
0810 18:12:23.278907 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:23.278920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:23.278943 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:12:23.279025 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:23.279030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:23.279249 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:23.279814 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:12:23.279973 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.281173 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.282982 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.283844 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.284590 (+ 746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.284743 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:23.284794 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 18:12:23.284882 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":121,"threads_started":1}
I20260810 18:12:23.285182 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.278622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:44327 (local address 127.0.0.1:49626)
0810 18:12:23.279529 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:23.279561 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:23.279817 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:23.280204 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:23.280207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:23.280225 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:12:23.280326 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.280334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.281242 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.281246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.282946 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.282958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.284152 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.284155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.284256 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.284277 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 18:12:23.284338 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 18:12:23.284399 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":227,"thread_start_us":85,"threads_started":1}
I20260810 18:12:23.287846 8318 tablet_service.cc:1467] Processing CreateTablet for tablet d852ba7fe74842fd975f7d1f270bd9db (DEFAULT_TABLE table=test-table [id=16a009cedc374426b586b5f9b9a6c8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:12:23.287861 8538 tablet_service.cc:1467] Processing CreateTablet for tablet d852ba7fe74842fd975f7d1f270bd9db (DEFAULT_TABLE table=test-table [id=16a009cedc374426b586b5f9b9a6c8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:12:23.287861 8791 tablet_service.cc:1467] Processing CreateTablet for tablet d852ba7fe74842fd975f7d1f270bd9db (DEFAULT_TABLE table=test-table [id=16a009cedc374426b586b5f9b9a6c8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:12:23.288825 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d852ba7fe74842fd975f7d1f270bd9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:12:23.288879 8538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d852ba7fe74842fd975f7d1f270bd9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:12:23.288887 8791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d852ba7fe74842fd975f7d1f270bd9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:12:23.295619 8929 tablet_bootstrap.cc:492] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8: Bootstrap starting.
I20260810 18:12:23.296300 8933 tablet_bootstrap.cc:492] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: Bootstrap starting.
I20260810 18:12:23.297706 8929 tablet_bootstrap.cc:654] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8: Neither blocks nor log segments found. Creating new log.
I20260810 18:12:23.298522 8933 tablet_bootstrap.cc:654] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: Neither blocks nor log segments found. Creating new log.
I20260810 18:12:23.298766 8929 log.cc:824] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8: Log is configured to *not* fsync() on all Append() calls
I20260810 18:12:23.299259 8933 log.cc:824] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:12:23.299397 8930 tablet_bootstrap.cc:492] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: Bootstrap starting.
I20260810 18:12:23.301030 8929 tablet_bootstrap.cc:492] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8: No bootstrap required, opened a new log
I20260810 18:12:23.301244 8929 ts_tablet_manager.cc:1397] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:12:23.301532 8933 tablet_bootstrap.cc:492] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: No bootstrap required, opened a new log
I20260810 18:12:23.301571 8930 tablet_bootstrap.cc:654] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: Neither blocks nor log segments found. Creating new log.
I20260810 18:12:23.301707 8933 ts_tablet_manager.cc:1397] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:12:23.302735 8930 log.cc:824] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:12:23.304783 8930 tablet_bootstrap.cc:492] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: No bootstrap required, opened a new log
I20260810 18:12:23.304740 8933 raft_consensus.cc:348] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } }
I20260810 18:12:23.304970 8930 ts_tablet_manager.cc:1397] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:12:23.305009 8933 raft_consensus.cc:374] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:12:23.305065 8933 raft_consensus.cc:729] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031b7b8c983d469486f3230632cc3d3a, State: Initialized, Role: FOLLOWER
I20260810 18:12:23.305032 8929 raft_consensus.cc:348] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } }
I20260810 18:12:23.305305 8929 raft_consensus.cc:374] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:12:23.305361 8929 raft_consensus.cc:729] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5e67145ee04bdd90ec14c02a0482e8, State: Initialized, Role: FOLLOWER
I20260810 18:12:23.305526 8933 consensus_queue.cc:260] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } }
I20260810 18:12:23.305787 8929 consensus_queue.cc:260] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } }
I20260810 18:12:23.306344 8668 heartbeater.cc:503] Master 127.23.65.126:45551 was elected leader, sending a full tablet report...
I20260810 18:12:23.306557 8933 ts_tablet_manager.cc:1428] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:12:23.307967 8929 ts_tablet_manager.cc:1428] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 18:12:23.308328 8421 heartbeater.cc:503] Master 127.23.65.126:45551 was elected leader, sending a full tablet report...
I20260810 18:12:23.308681 8930 raft_consensus.cc:348] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } }
I20260810 18:12:23.308887 8930 raft_consensus.cc:374] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:12:23.308921 8930 raft_consensus.cc:729] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ea3a6b2fda40e88848a2ecf673913a, State: Initialized, Role: FOLLOWER
I20260810 18:12:23.309284 8930 consensus_queue.cc:260] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } }
I20260810 18:12:23.309984 8897 heartbeater.cc:503] Master 127.23.65.126:45551 was elected leader, sending a full tablet report...
I20260810 18:12:23.310341 8930 ts_tablet_manager.cc:1428] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:12:23.452922 8945 raft_consensus.cc:482] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:12:23.453137 8945 raft_consensus.cc:504] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } }
I20260810 18:12:23.454355 8945 leader_election.cc:288] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70ea3a6b2fda40e88848a2ecf673913a (127.23.65.67:39785), 5a5e67145ee04bdd90ec14c02a0482e8 (127.23.65.65:46483)
I20260810 18:12:23.458375 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.454630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:39785 (local address 127.23.65.66:50409)
0810 18:12:23.454787 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:23.454803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:23.454911 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:23.455176 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:23.455179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:23.455199 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:12:23.455315 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.455324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.456260 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.456264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.457248 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.457259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.457967 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.457971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.458073 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.458100 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:12:23.458169 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:12:23.458236 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":4}
I20260810 18:12:23.458405 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.454747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:50409 (local address 127.23.65.67:39785)
0810 18:12:23.454888 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:23.454892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:23.454910 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:12:23.455009 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:23.455011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:23.455067 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:23.455155 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:12:23.455449 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.456153 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.457382 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.457863 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.457953 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.457995 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:23.458188 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 18:12:23.458257 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:12:23.458578 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.454647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:46483 (local address 127.23.65.66:43389)
0810 18:12:23.455198 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:23.455209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:12:23.455315 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:12:23.455580 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:12:23.455583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:12:23.455602 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:12:23.455731 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.455737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.456566 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.456569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.457511 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:12:23.457522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.458195 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.458198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:12:23.458293 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.458317 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:12:23.458392 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:12:23.458454 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":181,"thread_start_us":132,"threads_started":1}
I20260810 18:12:23.458613 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.454876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:43389 (local address 127.23.65.65:46483)
0810 18:12:23.455002 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:23.455007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:23.455295 (+ 288us) server_negotiation.cc:407] Connection header received
0810 18:12:23.455417 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:23.455421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:23.455472 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:23.455525 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:12:23.455866 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.456454 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.457637 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.458071 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.458186 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.458232 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:23.458424 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 18:12:23.458500 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:12:23.459070 8816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d852ba7fe74842fd975f7d1f270bd9db" candidate_uuid: "031b7b8c983d469486f3230632cc3d3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ea3a6b2fda40e88848a2ecf673913a" is_pre_election: true
I20260810 18:12:23.459347 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d852ba7fe74842fd975f7d1f270bd9db" candidate_uuid: "031b7b8c983d469486f3230632cc3d3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" is_pre_election: true
I20260810 18:12:23.459399 8816 raft_consensus.cc:2454] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031b7b8c983d469486f3230632cc3d3a in term 0.
I20260810 18:12:23.459657 8342 raft_consensus.cc:2454] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031b7b8c983d469486f3230632cc3d3a in term 0.
I20260810 18:12:23.459931 8463 leader_election.cc:302] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031b7b8c983d469486f3230632cc3d3a, 70ea3a6b2fda40e88848a2ecf673913a; no voters:
I20260810 18:12:23.460211 8945 raft_consensus.cc:2781] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:12:23.460299 8945 raft_consensus.cc:482] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:12:23.460330 8945 raft_consensus.cc:3037] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:12:23.461352 8945 raft_consensus.cc:504] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } }
I20260810 18:12:23.461650 8945 leader_election.cc:288] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [CANDIDATE]: Term 1 election: Requested vote from peers 70ea3a6b2fda40e88848a2ecf673913a (127.23.65.67:39785), 5a5e67145ee04bdd90ec14c02a0482e8 (127.23.65.65:46483)
I20260810 18:12:23.462033 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d852ba7fe74842fd975f7d1f270bd9db" candidate_uuid: "031b7b8c983d469486f3230632cc3d3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5e67145ee04bdd90ec14c02a0482e8"
I20260810 18:12:23.462154 8342 raft_consensus.cc:3037] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:12:23.463164 8342 raft_consensus.cc:2454] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031b7b8c983d469486f3230632cc3d3a in term 1.
I20260810 18:12:23.463423 8816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d852ba7fe74842fd975f7d1f270bd9db" candidate_uuid: "031b7b8c983d469486f3230632cc3d3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ea3a6b2fda40e88848a2ecf673913a"
I20260810 18:12:23.463469 8460 leader_election.cc:302] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031b7b8c983d469486f3230632cc3d3a, 5a5e67145ee04bdd90ec14c02a0482e8; no voters:
I20260810 18:12:23.463581 8816 raft_consensus.cc:3037] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:12:23.463663 8945 raft_consensus.cc:2781] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:12:23.463891 8945 raft_consensus.cc:686] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 1 LEADER]: Becoming Leader. State: Replica: 031b7b8c983d469486f3230632cc3d3a, State: Running, Role: LEADER
I20260810 18:12:23.464279 8945 consensus_queue.cc:237] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } }
I20260810 18:12:23.464819 8816 raft_consensus.cc:2454] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031b7b8c983d469486f3230632cc3d3a in term 1.
I20260810 18:12:23.467335 8082 catalog_manager.cc:5495] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 031b7b8c983d469486f3230632cc3d3a (127.23.65.66). New cstate: current_term: 1 leader_uuid: "031b7b8c983d469486f3230632cc3d3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031b7b8c983d469486f3230632cc3d3a" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44327 } health_report { overall_health: HEALTHY } } }
W20260810 18:12:23.485212 8898 tablet.cc:2369] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:12:23.512269 8669 tablet.cc:2369] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:12:23.517969 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.514801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.23.65.66:44327)
0810 18:12:23.514931 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:12:23.514935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:12:23.514947 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:12:23.515021 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:12:23.515024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:12:23.515077 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:12:23.515166 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:12:23.515426 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.515943 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.516743 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:12:23.516948 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:12:23.517011 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:12:23.517228 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:12:23.517637 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:12:23.517724 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 18:12:23.517752 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 18:12:23.517799 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:12:23.523612 8342 raft_consensus.cc:1261] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8 [term 1 FOLLOWER]: Refusing update from remote peer 031b7b8c983d469486f3230632cc3d3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:12:23.523618 8816 raft_consensus.cc:1261] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [term 1 FOLLOWER]: Refusing update from remote peer 031b7b8c983d469486f3230632cc3d3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:12:23.524339 8952 consensus_queue.cc:1035] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a5e67145ee04bdd90ec14c02a0482e8" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 46483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:12:23.524515 8945 consensus_queue.cc:1035] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ea3a6b2fda40e88848a2ecf673913a" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:12:23.541561 8080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:12:23.541730 8080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:12:23.544118 8080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72a568282b7346c6b550ce567eb9f59d: Sending DeleteTablet for 3 replicas of tablet d852ba7fe74842fd975f7d1f270bd9db
I20260810 18:12:23.545035 8318 tablet_service.cc:1514] Processing DeleteTablet for tablet d852ba7fe74842fd975f7d1f270bd9db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:12:23 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:44362
I20260810 18:12:23.545059 8791 tablet_service.cc:1514] Processing DeleteTablet for tablet d852ba7fe74842fd975f7d1f270bd9db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:12:23 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:58770
I20260810 18:12:23.545137 8538 tablet_service.cc:1514] Processing DeleteTablet for tablet d852ba7fe74842fd975f7d1f270bd9db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:12:23 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:49626
I20260810 18:12:23.545598 8980 tablet_replica.cc:330] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8: stopping tablet replica
I20260810 18:12:23.546186 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 8203
I20260810 18:12:23.546150 8980 raft_consensus.cc:2229] T d852ba7fe74842fd975f7d1f270bd9db P 5a5e67145ee04bdd90ec14c02a0482e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:12:23.545856 8979 tablet_replica.cc:330] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: stopping tablet replica
I20260810 18:12:23.546897 8979 raft_consensus.cc:2229] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:12:23.547269 8979 raft_consensus.cc:2258] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:12:23.554929 8981 tablet_replica.cc:330] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: stopping tablet replica
I20260810 18:12:23.555262 8981 raft_consensus.cc:2229] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:12:23.555595 8981 raft_consensus.cc:2258] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:12:23.557180 8979 ts_tablet_manager.cc:1905] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:12:23.558454 8981 ts_tablet_manager.cc:1905] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:12:23.560201 8979 ts_tablet_manager.cc:1918] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:12:23.560390 8979 log.cc:1196] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/d852ba7fe74842fd975f7d1f270bd9db
I20260810 18:12:23.560788 8979 ts_tablet_manager.cc:1939] T d852ba7fe74842fd975f7d1f270bd9db P 031b7b8c983d469486f3230632cc3d3a: Deleting consensus metadata
I20260810 18:12:23.561060 8981 ts_tablet_manager.cc:1918] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:12:23.561160 8981 log.cc:1196] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/d852ba7fe74842fd975f7d1f270bd9db
I20260810 18:12:23.561477 8981 ts_tablet_manager.cc:1939] T d852ba7fe74842fd975f7d1f270bd9db P 70ea3a6b2fda40e88848a2ecf673913a: Deleting consensus metadata
I20260810 18:12:23.562280 8057 catalog_manager.cc:4841] TS 031b7b8c983d469486f3230632cc3d3a (127.23.65.66:44327): tablet d852ba7fe74842fd975f7d1f270bd9db (table test-table [id=16a009cedc374426b586b5f9b9a6c8a2]) successfully deleted
I20260810 18:12:23.562834 8060 catalog_manager.cc:4841] TS 70ea3a6b2fda40e88848a2ecf673913a (127.23.65.67:39785): tablet d852ba7fe74842fd975f7d1f270bd9db (table test-table [id=16a009cedc374426b586b5f9b9a6c8a2]) successfully deleted
W20260810 18:12:23.564515 8057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:46483 (error 108)
I20260810 18:12:23.564776 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 8427
I20260810 18:12:23.582840 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:12:23.582446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:46483 (local address 127.0.0.1:44378)
0810 18:12:23.582640 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 18:12:23.582730 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260810 18:12:23.583180 8057 catalog_manager.cc:4584] TS 5a5e67145ee04bdd90ec14c02a0482e8 (127.23.65.65:46483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d852ba7fe74842fd975f7d1f270bd9db: Network error: Client connection negotiation failed: client connection to 127.23.65.65:46483: connect: Connection refused (error 111)
I20260810 18:12:23.592586 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 8682
I20260810 18:12:23.602520 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 8034
2026-08-10T18:12:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2597 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:12:23 dist-test-slave-jpj1 krb5kdc[8985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:12:23 dist-test-slave-jpj1 krb5kdc[8985](info): set up 2 sockets
Aug 10 18:12:23 dist-test-slave-jpj1 krb5kdc[8985](info): commencing operation
krb5kdc: starting...
W20260810 18:12:24.769289 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:12:24 dist-test-slave-jpj1 krb5kdc[8985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:12:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.65.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.65.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/rangeradmin_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/rangeradmin_127.23.65.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.65.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.65.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/rangerlookup_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/rangerlookup_127.23.65.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.65.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.65.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/HTTP_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/HTTP_127.23.65.84@KRBTEST.COM.keytab.
I20260810 18:12:24.938045 23813 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmz0U2F/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/postgres -l logfile start
2026-08-10 18:12:27.333 UTC [9058] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 18:12:27.334 UTC [9058] LOG: listening on IPv4 address "127.23.65.84", port 49045
2026-08-10 18:12:27.337 UTC [9058] LOG: listening on Unix socket "/tmp/.s.PGSQL.49045"
2026-08-10 18:12:27.366 UTC [9064] LOG: database system was shut down at 2026-08-10 18:12:26 UTC
2026-08-10 18:12:27.370 UTC [9058] LOG: database system is ready to accept connections
I20260810 18:12:28.426673 23813 mini_postgres.cc:96] Postgres bound to 49045
2026-08-10 18:12:28.434 UTC [9077] FATAL: database "slave" does not exist
127.23.65.84:49045 - accepting connections
I20260810 18:12:28.435468 23813 mini_ranger.cc:161] Starting Ranger...
I20260810 18:12:28.450809 23813 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 18:12:28.872812 23813 mini_ranger.cc:87] Created ranger Postgres database
I20260810 18:12:28.872929 23813 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-admin
2026-08-10 18:12:29,200 [I] DB FLAVOR :POSTGRES
2026-08-10 18:12:29,200 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:12:29,200 [I] Checking connection..
2026-08-10 18:12:29,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 18:12:29,470 [I] Checking connection passed.
2026-08-10 18:12:29,471 [I] --------- Verifying version history table ---------
2026-08-10 18:12:29,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:12:29,731 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:12:29,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:12:29,972 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:12:29,973 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 18:12:29,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 18:12:30.465 UTC [9184] WARNING: there is no transaction in progress
2026-08-10 18:12:30,481 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 18:12:30,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:12:30,969 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 18:12:30,970 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:12:30,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 18:12:31,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 18:12:31,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jpj1.c.gcp-upstream.internal','N') ;"
2026-08-10 18:12:32,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:12:32,384 [I] Table x_portal_user does not exist in database ranger
2026-08-10 18:12:32,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:12:32,627 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 18:12:32,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:12:32,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:12:33,124 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 18:12:33,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 18:12:34.039 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.055 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.070 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.085 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.101 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.121 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.175 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.182 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.195 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.205 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.214 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.228 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.238 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.244 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.251 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34.257 UTC [9372] WARNING: there is no transaction in progress
2026-08-10 18:12:34,849 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 18:12:34,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 18:12:35,089 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 18:12:35,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:12:35,353 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 18:12:35,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:12:35,603 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 18:12:35,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:12:35,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:12:36,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jpj1.c.gcp-upstream.internal';"
2026-08-10 18:12:36,362 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 18:12:36,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 18:12:36,619 [I] DB_PATCHES have already been applied
I20260810 18:12:36.625968 23813 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmz0U2F/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 18:12:36.626039 23813 mini_ranger.cc:193] Using host: 127.23.65.84
I20260810 18:12:36.630496 23813 mini_ranger.cc:227] Ranger admin URL: http://127.23.65.84:51515
Aug 10, 2026 6:12:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:12:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:12:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:12:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:12:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:12:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:12:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 18:12:37.890970 23813 mini_ranger.cc:160] Time spent starting Ranger: real 9.456s user 0.000s sys 0.012s
Aug 10, 2026 6:12:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:12:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 6:12:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:12:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:12:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 18:12:58.795748 23813 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.65.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.65.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/rangerkms_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/rangerkms_127.23.65.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.65.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.65.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.65.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/HTTP_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.65.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/HTTP_127.23.65.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 18:12:58 dist-test-slave-jpj1 krb5kdc[8985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385578, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 18:12:58.912150 23813 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 18:12:58.924580 23813 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 18:12:59.916409 23813 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 18:12:59.916497 23813 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-kms
I20260810 18:12:59.916603 23813 mini_ranger_kms.cc:227] Using postgres at 127.23.65.84:49045
2026-08-10 18:13:00,464 [I] DB FLAVOR :POSTGRES
2026-08-10 18:13:00,465 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:13:00,465 [I] Checking connection
2026-08-10 18:13:00,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 18:13:00,788 [I] connection success
2026-08-10 18:13:00,788 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 18:13:00,788 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 18:13:00,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 18:13:01,033 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 18:13:01,033 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:13:01,033 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 18:13:01,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:49045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 18:13:01,298 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 18:13:01.786160 23813 mini_ranger_kms.cc:310] Created kms service
I20260810 18:13:01.982086 23813 mini_ranger_kms.cc:326] Created kudu user
I20260810 18:13:02.027411 23813 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 18:13:02 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385578, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:02.425252 23813 mini_ranger_kms.cc:384] Added ranger policy
I20260810 18:13:02.425418 23813 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 18:13:02.425438 23813 mini_ranger_kms.cc:240] Using host: 127.23.65.84
I20260810 18:13:02.430357 23813 mini_ranger_kms.cc:276] Ranger KMS PID: 9684
I20260810 18:13:02.430505 23813 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.65.84:53867
Aug 10, 2026 6:13:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:13:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:13:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:13:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:13:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:13:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 18:13:04 dist-test-slave-jpj1 krb5kdc[8985](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385584, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.65.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:05.011008 23813 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.099s user 0.016s sys 0.512s
I20260810 18:13:05.011142 23813 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 18:13:05.011255 23813 mini_ranger_kms.cc:401] 127.23.65.84:53867/kms/v1/keys
Aug 10 18:13:05 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385584, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.65.84@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
Aug 10 18:13:05 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385584, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.65.84@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
Aug 10 18:13:08 dist-test-slave-jpj1 krb5kdc[8985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:13:08.422833 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40041
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:40041
--ranger_config_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:53867/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:13:08.577085 9770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:08.577369 9770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:08.577430 9770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:08.581533 9770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:13:08.581661 9770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:08.581732 9770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:08.581766 9770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:13:08.581796 9770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:13:08.587671 9770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.65.84:53867/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:40041
--ranger_config_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:08.588729 9770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:08.589830 9770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:08.600023 9780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:08.599957 9778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:08.600209 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:08.600587 9770 server_base.cc:1034] running on GCE node
I20260810 18:13:08.601270 9770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:08.602447 9770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:08.603611 9770 hybrid_clock.cc:648] HybridClock initialized: now 1786385588603588 us; error 34 us; skew 500 ppm
Aug 10 18:13:08 dist-test-slave-jpj1 krb5kdc[8985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:08.607304 9770 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:13:08.608345 9770 webserver.cc:460] Webserver started at http://127.23.65.126:39441/ using document root <none> and password file <none>
I20260810 18:13:08.608887 9770 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:08.608946 9770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:13:08.609157 9770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:13:08 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:08.673993 9770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "466fd9462dea4ca195daa14c0fac0534"
format_stamp: "Formatted at 2026-08-10 18:13:08 on dist-test-slave-jpj1"
server_key: "3803e1ae01d4b50ccaf443e6ea70b240"
server_key_iv: "8b86a48a9408d34e43f00b0121555db8"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:08.674682 9770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "466fd9462dea4ca195daa14c0fac0534"
format_stamp: "Formatted at 2026-08-10 18:13:08 on dist-test-slave-jpj1"
server_key: "3803e1ae01d4b50ccaf443e6ea70b240"
server_key_iv: "8b86a48a9408d34e43f00b0121555db8"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:08.679526 9770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 18:13:08.697945 9790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:08.699373 9770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:13:08.699497 9770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "466fd9462dea4ca195daa14c0fac0534"
format_stamp: "Formatted at 2026-08-10 18:13:08 on dist-test-slave-jpj1"
server_key: "3803e1ae01d4b50ccaf443e6ea70b240"
server_key_iv: "8b86a48a9408d34e43f00b0121555db8"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:08.699667 9770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:08.737305 9770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:08.740595 9770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:08.740794 9770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:08.755450 9770 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:40041
I20260810 18:13:08.755558 9880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:40041 every 8 connection(s)
I20260810 18:13:08.756819 9770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:13:08.760262 9881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:13:08.762305 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 9770
I20260810 18:13:08.762449 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:13:08.762771 23813 external_mini_cluster.cc:1428] Setting key 1229cb842bfe9f26e0de69ccc05a986a
I20260810 18:13:08.768911 9881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534: Bootstrap starting.
I20260810 18:13:08.772400 9881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534: Neither blocks nor log segments found. Creating new log.
I20260810 18:13:08.773993 9881 log.cc:824] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:13:08 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:13:08.776379 9881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534: No bootstrap required, opened a new log
I20260810 18:13:08.781690 9881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466fd9462dea4ca195daa14c0fac0534" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40041 } }
I20260810 18:13:08.781967 9881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:13:08.781996 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:08.764388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.23.65.126:40041)
0810 18:13:08.765094 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:08.765149 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:08.765197 (+ 48us) server_negotiation.cc:407] Connection header received
0810 18:13:08.766720 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:08.766746 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:08.767181 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:08.767575 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:08.768493 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:08.770209 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:08.771490 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:08.772285 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:08.772437 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:08.775557 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:08.775565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:08.775582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:08.775614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:08.778736 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:08.779254 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:08.779260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:08.779265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:08.779403 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:08.779694 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:08.779697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:08.779699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:08.780221 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:08.780421 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:08.780707 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 18:13:08.780995 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":229,"threads_started":1}
I20260810 18:13:08.782075 9881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 466fd9462dea4ca195daa14c0fac0534, State: Initialized, Role: FOLLOWER
I20260810 18:13:08.782606 9881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466fd9462dea4ca195daa14c0fac0534" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40041 } }
I20260810 18:13:08.782752 9881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:13:08.782832 9881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:13:08.782930 9881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:13:08.784356 9881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466fd9462dea4ca195daa14c0fac0534" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40041 } }
I20260810 18:13:08.784709 9881 leader_election.cc:302] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 466fd9462dea4ca195daa14c0fac0534; no voters:
I20260810 18:13:08.785086 9881 leader_election.cc:288] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:13:08.785295 9891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:13:08.785669 9891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [term 1 LEADER]: Becoming Leader. State: Replica: 466fd9462dea4ca195daa14c0fac0534, State: Running, Role: LEADER
I20260810 18:13:08.786026 9891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466fd9462dea4ca195daa14c0fac0534" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40041 } }
I20260810 18:13:08.786463 9881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:13:08.787357 9891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 466fd9462dea4ca195daa14c0fac0534. Latest consensus state: current_term: 1 leader_uuid: "466fd9462dea4ca195daa14c0fac0534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466fd9462dea4ca195daa14c0fac0534" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40041 } } }
I20260810 18:13:08.787581 9891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [sys.catalog]: This master's current role is: LEADER
I20260810 18:13:08.787953 9894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "466fd9462dea4ca195daa14c0fac0534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466fd9462dea4ca195daa14c0fac0534" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40041 } } }
I20260810 18:13:08.788054 9894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534 [sys.catalog]: This master's current role is: LEADER
I20260810 18:13:08.790163 9881 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 18:13:08.790233 9903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:13:08.799657 9903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:13:08.818287 9903 catalog_manager.cc:1347] Generated new cluster ID: ca67fa56a02847c5b98383fdd5459e13
I20260810 18:13:08.818406 9903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:13:08.830770 9903 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:13:08.832209 9903 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:13:08.837700 9903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534: Generated new TSK 0
I20260810 18:13:08.838611 9903 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:13:09.707662 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:53867/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40041
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:13:09.909188 9963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:09.909541 9963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:09.909621 9963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:09.915519 9963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:09.915709 9963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:09.915858 9963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:13:09.923735 9963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.65.84:53867/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:09.925213 9963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:09.926576 9963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:09.939250 9974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:09.939469 9979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:09.939303 9973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:09.939838 9963 server_base.cc:1034] running on GCE node
I20260810 18:13:09.940346 9963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:09.941145 9963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:09.942344 9963 hybrid_clock.cc:648] HybridClock initialized: now 1786385589942317 us; error 45 us; skew 500 ppm
Aug 10 18:13:09 dist-test-slave-jpj1 krb5kdc[8985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:09.945755 9963 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:13:09.947002 9963 webserver.cc:460] Webserver started at http://127.23.65.65:45425/ using document root <none> and password file <none>
I20260810 18:13:09.947785 9963 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:09.947889 9963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:13:09.948249 9963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:13:09 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:09.970904 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "5980fa6c34394a9bac8e35454957333a"
format_stamp: "Formatted at 2026-08-10 18:13:09 on dist-test-slave-jpj1"
server_key: "8b6844b1dbc8f8e56d4181bec8e4d9c0"
server_key_iv: "3c25a9982e80b45e770dedda20f5a7d2"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:09.971678 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "5980fa6c34394a9bac8e35454957333a"
format_stamp: "Formatted at 2026-08-10 18:13:09 on dist-test-slave-jpj1"
server_key: "8b6844b1dbc8f8e56d4181bec8e4d9c0"
server_key_iv: "3c25a9982e80b45e770dedda20f5a7d2"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:09.977329 9963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 18:13:10.000316 9994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:10.002122 9963 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260810 18:13:10.002266 9963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "5980fa6c34394a9bac8e35454957333a"
format_stamp: "Formatted at 2026-08-10 18:13:09 on dist-test-slave-jpj1"
server_key: "8b6844b1dbc8f8e56d4181bec8e4d9c0"
server_key_iv: "3c25a9982e80b45e770dedda20f5a7d2"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:10.002413 9963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:10.039093 9963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:10.043231 9963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:10.043548 9963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:10.044412 9963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:13:10.045722 9963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:13:10.045845 9963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 18:13:10.045993 9963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:13:10.046087 9963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:10.065058 9963 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:34035
I20260810 18:13:10.066442 9963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:13:10.068395 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 9963
I20260810 18:13:10.068537 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:13:10.068876 23813 external_mini_cluster.cc:1428] Setting key a1426e9bf1e2d2cf476bab94e2cef3ea
Aug 10 18:13:10 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:13:10.089608 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.071172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:52013 (local address 127.23.65.126:40041)
0810 18:13:10.073707 (+ 2535us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:10.073711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:10.073727 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:13:10.073770 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:10.073773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:10.073824 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:10.073917 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:10.075697 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.076457 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.080715 (+ 4258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.081116 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.081202 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.084793 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:10.084797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:10.084799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:10.084844 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:10.086836 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:10.087615 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:10.087619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:10.087622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:10.087725 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:10.088178 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:10.088191 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:10.088193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:10.088431 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:10.088519 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:10.089256 (+ 737us) server_negotiation.cc:299] Negotiation successful
0810 18:13:10.089397 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2455,"thread_start_us":135,"threads_started":1}
I20260810 18:13:10.091974 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.070719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40041 (local address 127.23.65.65:52013)
0810 18:13:10.071825 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:10.071889 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:10.073264 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:10.074258 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:10.074267 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:10.074762 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:13:10.075504 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.075526 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.076621 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.076625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.080543 (+ 3918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.080556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.081498 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.081503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.081719 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.082493 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:13:10.082515 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:13:10.084562 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:13:10.087056 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:10.087063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:10.087083 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:10.087439 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:10.087851 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:10.087855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:10.087857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:10.088054 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:10.088595 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:13:10.088604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:13:10.088945 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 18:13:10.090100 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0810 18:13:10.090518 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":261,"threads_started":1}
I20260810 18:13:10.092478 10197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:34035 every 8 connection(s)
I20260810 18:13:10.094128 10198 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40041
I20260810 18:13:10.094482 10198 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:10.095108 10198 heartbeater.cc:511] Master 127.23.65.126:40041 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
I20260810 18:13:10.097263 9820 ts_manager.cc:194] Registered new tserver with Master: 5980fa6c34394a9bac8e35454957333a (127.23.65.65:34035)
I20260810 18:13:10.098929 9820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:52013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:13:10.119122 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:53867/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40041
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:13:10 dist-test-slave-jpj1 krb5kdc[8985](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 18:13:10.310832 10212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:10.311343 10212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:10.311475 10212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:10.317889 10212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:10.318063 10212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:10.318205 10212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:13:10.326584 10212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.65.84:53867/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:10.328431 10212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:10.329918 10212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:10.350021 10221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:10.350687 10225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:10.351727 10220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:10.351989 10212 server_base.cc:1034] running on GCE node
I20260810 18:13:10.352488 10212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:10.353328 10212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:10.354489 10212 hybrid_clock.cc:648] HybridClock initialized: now 1786385590354426 us; error 82 us; skew 500 ppm
Aug 10 18:13:10 dist-test-slave-jpj1 krb5kdc[8985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:10.357704 10212 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:13:10.358693 10212 webserver.cc:460] Webserver started at http://127.23.65.66:41749/ using document root <none> and password file <none>
I20260810 18:13:10.359256 10212 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:10.359324 10212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:13:10.359584 10212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:13:10 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:10.374185 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "6cc54c386b5342e1adac607c3f436cb1"
format_stamp: "Formatted at 2026-08-10 18:13:10 on dist-test-slave-jpj1"
server_key: "d50e729952f60e7a7740d4d690310c1d"
server_key_iv: "2e8e1c1ac4773a2e852932bf6b936ddc"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:10.375363 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "6cc54c386b5342e1adac607c3f436cb1"
format_stamp: "Formatted at 2026-08-10 18:13:10 on dist-test-slave-jpj1"
server_key: "d50e729952f60e7a7740d4d690310c1d"
server_key_iv: "2e8e1c1ac4773a2e852932bf6b936ddc"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:10.379374 10212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:13:10.399919 10234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:10.401367 10212 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260810 18:13:10.401497 10212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "6cc54c386b5342e1adac607c3f436cb1"
format_stamp: "Formatted at 2026-08-10 18:13:10 on dist-test-slave-jpj1"
server_key: "d50e729952f60e7a7740d4d690310c1d"
server_key_iv: "2e8e1c1ac4773a2e852932bf6b936ddc"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:10.401629 10212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:10.425289 10212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:10.429549 10212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:10.429883 10212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:10.430900 10212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:13:10.432245 10212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:13:10.432369 10212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:10.432497 10212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:13:10.432567 10212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:10.454702 10212 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:38471
I20260810 18:13:10.456323 10212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:13:10.458462 10414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:38471 every 8 connection(s)
I20260810 18:13:10.464212 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 10212
I20260810 18:13:10.464372 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:13:10.464744 23813 external_mini_cluster.cc:1428] Setting key ff2458b378dc24505d6afefcba1b2637
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Aug 10 18:13:10 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:13:10.493100 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.462255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:35855 (local address 127.23.65.126:40041)
0810 18:13:10.462386 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:10.462391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:10.468123 (+ 5732us) server_negotiation.cc:407] Connection header received
0810 18:13:10.474681 (+ 6558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:10.474686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:10.474751 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:10.474811 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:10.476937 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.477768 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.479422 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.479759 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.479852 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.488726 (+ 8874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:10.488731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:10.488734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:10.488768 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:10.490574 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:10.491273 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:10.491277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:10.491279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:10.491369 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:10.491829 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:10.491833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:10.491835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:10.492071 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:10.492143 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:10.492793 (+ 650us) server_negotiation.cc:299] Negotiation successful
0810 18:13:10.492928 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:13:10.494266 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.460991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40041 (local address 127.23.65.66:35855)
0810 18:13:10.467949 (+ 6958us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:10.468019 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:10.474342 (+ 6323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:10.475311 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:10.475322 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:10.475882 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:13:10.476746 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.476769 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.477941 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.477944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.479298 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.479308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.481168 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.481171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.481380 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.486242 (+ 4862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:13:10.486265 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:13:10.488459 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:13:10.490742 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:10.490749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:10.490769 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:10.491145 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:10.491480 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:10.491483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:10.491485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:10.491707 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:10.492201 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:13:10.492207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:13:10.492456 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 18:13:10.492718 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 18:13:10.493052 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6668,"thread_start_us":200,"threads_started":1}
I20260810 18:13:10.495633 10415 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40041
I20260810 18:13:10.495959 10415 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:10.496539 10415 heartbeater.cc:511] Master 127.23.65.126:40041 requested a full tablet report, sending...
I20260810 18:13:10.497618 9820 ts_manager.cc:194] Registered new tserver with Master: 6cc54c386b5342e1adac607c3f436cb1 (127.23.65.66:38471)
I20260810 18:13:10.498311 9820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:35855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:13:10.507342 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:53867/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40041
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:13:10.706200 10426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:10.706465 10426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:10.706521 10426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:10.710935 10426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:10.711010 10426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:10.711089 10426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:13:10.717552 10426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.65.84:53867/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:10.718760 10426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:10.719867 10426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:10.729979 10436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:10.730165 10434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:10.730944 10433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:10.731009 10426 server_base.cc:1034] running on GCE node
I20260810 18:13:10.731527 10426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:10.732333 10426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:10.733822 10426 hybrid_clock.cc:648] HybridClock initialized: now 1786385590733756 us; error 80 us; skew 500 ppm
Aug 10 18:13:10 dist-test-slave-jpj1 krb5kdc[8985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:10.737458 10426 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:13:10.738655 10426 webserver.cc:460] Webserver started at http://127.23.65.67:41307/ using document root <none> and password file <none>
I20260810 18:13:10.739377 10426 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:10.739490 10426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:13:10.739776 10426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:13:10 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:10.762018 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "8612db7195764567b4d00afaa161fc44"
format_stamp: "Formatted at 2026-08-10 18:13:10 on dist-test-slave-jpj1"
server_key: "110ef70c3ffe2c3297374f8140274d4f"
server_key_iv: "5c5f152cefc8340f3dfee9076eba9156"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:10.762975 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "8612db7195764567b4d00afaa161fc44"
format_stamp: "Formatted at 2026-08-10 18:13:10 on dist-test-slave-jpj1"
server_key: "110ef70c3ffe2c3297374f8140274d4f"
server_key_iv: "5c5f152cefc8340f3dfee9076eba9156"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:10.772073 10426 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260810 18:13:10.797509 10445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:10.799491 10426 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260810 18:13:10.799796 10426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "8612db7195764567b4d00afaa161fc44"
format_stamp: "Formatted at 2026-08-10 18:13:10 on dist-test-slave-jpj1"
server_key: "110ef70c3ffe2c3297374f8140274d4f"
server_key_iv: "5c5f152cefc8340f3dfee9076eba9156"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:10.799991 10426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 10 18:13:10 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:10.869290 10426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:10.873597 10426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:10.873801 10426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:10.874675 10426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:13:10.875936 10426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:13:10.875994 10426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:10.876050 10426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:13:10.876072 10426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:10.890056 10426 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:35881
I20260810 18:13:10.890106 10636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:35881 every 8 connection(s)
I20260810 18:13:10.891302 10426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:13:10.893872 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 10426
I20260810 18:13:10.894011 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:13:10.894331 23813 external_mini_cluster.cc:1428] Setting key 3b24dd2615d40618bd1d65ab6a0d6765
Aug 10 18:13:10 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:13:10.921904 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.905978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:41661 (local address 127.23.65.126:40041)
0810 18:13:10.906117 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:10.906121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:10.908118 (+ 1997us) server_negotiation.cc:407] Connection header received
0810 18:13:10.909497 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:10.909501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:10.909559 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:10.909628 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:10.911348 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.912206 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.914061 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.914432 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.914510 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.917630 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:10.917634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:10.917637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:10.917666 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:10.919363 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:10.920018 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:10.920022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:10.920024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:10.920120 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:10.920528 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:10.920531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:10.920532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:10.920839 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:10.920930 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:10.921602 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 18:13:10.921729 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:13:10.923601 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.906612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40041 (local address 127.23.65.67:41661)
0810 18:13:10.907949 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:10.908014 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:10.909240 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:10.910074 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:10.910084 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:10.910485 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:13:10.911170 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.911190 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.912406 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.912410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.913909 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.913922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.914721 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.914725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.914926 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.915592 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:13:10.915613 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:13:10.917423 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:13:10.919506 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:10.919512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:10.919529 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:10.919896 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:10.920243 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:10.920246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:10.920248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:10.920426 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:10.920999 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:13:10.921006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:13:10.921326 (+ 320us) client_negotiation.cc:769] Sending connection context
0810 18:13:10.922136 (+ 810us) client_negotiation.cc:240] Negotiation successful
0810 18:13:10.922498 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":163,"threads_started":1}
I20260810 18:13:10.924935 10637 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40041
I20260810 18:13:10.925194 10637 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:10.925755 10637 heartbeater.cc:511] Master 127.23.65.126:40041 requested a full tablet report, sending...
I20260810 18:13:10.926785 9820 ts_manager.cc:194] Registered new tserver with Master: 8612db7195764567b4d00afaa161fc44 (127.23.65.67:35881)
I20260810 18:13:10.927569 9820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:41661
I20260810 18:13:10.935068 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:13:10.947202 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.937412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.23.65.126:40041)
0810 18:13:10.937522 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:10.937526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:10.938114 (+ 588us) server_negotiation.cc:407] Connection header received
0810 18:13:10.938301 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:10.938304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:10.938348 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:10.938394 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:10.939327 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.940171 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.941362 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.941694 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.941773 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.943590 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:10.943594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:10.943597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:10.943630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:10.945290 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:10.945777 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:10.945781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:10.945783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:10.945869 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:10.946223 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:10.946226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:10.946228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:10.946432 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:10.946530 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:10.946906 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 18:13:10.947026 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:13:10.951213 9820 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:13:10.954442 9820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:13:10.970860 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.963787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:38471 (local address 127.0.0.1:44488)
0810 18:13:10.965055 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:10.965091 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:10.965228 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:10.966065 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:10.966067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:10.966089 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:10.966266 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.966273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.967685 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.967688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.968977 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.968984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.970279 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.970283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.970441 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.970483 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 18:13:10.970585 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 18:13:10.970702 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":60,"spinlock_wait_cycles":1792,"thread_start_us":105,"threads_started":1}
I20260810 18:13:10.971452 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.963912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.23.65.66:38471)
0810 18:13:10.965633 (+ 1721us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:10.965650 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:10.965672 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:13:10.965744 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:10.965749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:10.965937 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:10.966085 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:10.966408 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.967493 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.969127 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.969973 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.970049 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.970183 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:10.971209 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0810 18:13:10.971314 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1619,"thread_start_us":70,"threads_started":1}
I20260810 18:13:10.971608 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.963884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:35881 (local address 127.0.0.1:46094)
0810 18:13:10.965289 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:10.965303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:10.965404 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:10.966044 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:10.966051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:10.966075 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:10.966266 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.966273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.968071 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.968075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.969330 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.969342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.971112 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.971116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.971239 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.971266 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:13:10.971344 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:13:10.971417 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":28,"spinlock_wait_cycles":1408,"thread_start_us":104,"threads_started":1}
I20260810 18:13:10.971978 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.964569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.23.65.67:35881)
0810 18:13:10.964897 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:10.964911 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:10.965413 (+ 502us) server_negotiation.cc:407] Connection header received
0810 18:13:10.965503 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:10.965508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:10.965717 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:10.965873 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:10.966395 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.967950 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.969506 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.970783 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.970892 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.971040 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:10.971719 (+ 679us) server_negotiation.cc:299] Negotiation successful
0810 18:13:10.971830 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
I20260810 18:13:10.974992 10296 tablet_service.cc:1467] Processing CreateTablet for tablet a446664d5d8a425abee3267f159f3931 (DEFAULT_TABLE table=test-table [id=f833db5ebf8b4ed387c5c2532f47f4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:13:10.975057 10488 tablet_service.cc:1467] Processing CreateTablet for tablet a446664d5d8a425abee3267f159f3931 (DEFAULT_TABLE table=test-table [id=f833db5ebf8b4ed387c5c2532f47f4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:13:10.975987 10488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a446664d5d8a425abee3267f159f3931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:13:10.975988 10296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a446664d5d8a425abee3267f159f3931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:13:10.980116 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.965276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:34035 (local address 127.0.0.1:59450)
0810 18:13:10.968172 (+ 2896us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:10.968186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:10.968273 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:10.968874 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:10.968877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:10.968892 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:10.968986 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.968991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.970289 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.970292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.971832 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:10.971843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.979643 (+ 7800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.979647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:10.979764 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.979793 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:13:10.979896 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 18:13:10.979979 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2823,"mutex_wait_us":25,"thread_start_us":89,"threads_started":1}
I20260810 18:13:10.980755 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:10.965691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.23.65.65:34035)
0810 18:13:10.966386 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:10.966392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:10.968365 (+ 1973us) server_negotiation.cc:407] Connection header received
0810 18:13:10.968440 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:10.968445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:10.968619 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:10.968749 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:10.969101 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.970181 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.971968 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:10.972867 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:10.972966 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:10.979830 (+ 6864us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:10.980500 (+ 670us) server_negotiation.cc:299] Negotiation successful
0810 18:13:10.980618 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":98,"threads_started":1}
I20260810 18:13:10.983922 10066 tablet_service.cc:1467] Processing CreateTablet for tablet a446664d5d8a425abee3267f159f3931 (DEFAULT_TABLE table=test-table [id=f833db5ebf8b4ed387c5c2532f47f4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:13:10.985114 10066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a446664d5d8a425abee3267f159f3931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:13:10.985743 10664 tablet_bootstrap.cc:492] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: Bootstrap starting.
I20260810 18:13:10.987273 10665 tablet_bootstrap.cc:492] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1: Bootstrap starting.
I20260810 18:13:10.991089 10665 tablet_bootstrap.cc:654] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1: Neither blocks nor log segments found. Creating new log.
I20260810 18:13:10.994350 10664 tablet_bootstrap.cc:654] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: Neither blocks nor log segments found. Creating new log.
I20260810 18:13:10.994319 10665 log.cc:824] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1: Log is configured to *not* fsync() on all Append() calls
I20260810 18:13:10.995316 10669 tablet_bootstrap.cc:492] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a: Bootstrap starting.
I20260810 18:13:10.995774 10664 log.cc:824] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: Log is configured to *not* fsync() on all Append() calls
I20260810 18:13:10.997815 10665 tablet_bootstrap.cc:492] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1: No bootstrap required, opened a new log
I20260810 18:13:10.998116 10665 ts_tablet_manager.cc:1397] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260810 18:13:10.998400 10664 tablet_bootstrap.cc:492] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: No bootstrap required, opened a new log
I20260810 18:13:10.998456 10669 tablet_bootstrap.cc:654] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a: Neither blocks nor log segments found. Creating new log.
I20260810 18:13:10.998608 10664 ts_tablet_manager.cc:1397] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20260810 18:13:10.999902 10669 log.cc:824] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:13:11.002305 10669 tablet_bootstrap.cc:492] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a: No bootstrap required, opened a new log
I20260810 18:13:11.002506 10669 ts_tablet_manager.cc:1397] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:13:11.005093 10664 raft_consensus.cc:348] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } }
I20260810 18:13:11.005404 10664 raft_consensus.cc:374] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:13:11.005465 10664 raft_consensus.cc:729] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8612db7195764567b4d00afaa161fc44, State: Initialized, Role: FOLLOWER
I20260810 18:13:11.005786 10665 raft_consensus.cc:348] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } }
I20260810 18:13:11.005995 10664 consensus_queue.cc:260] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } }
I20260810 18:13:11.006321 10665 raft_consensus.cc:374] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:13:11.006433 10665 raft_consensus.cc:729] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc54c386b5342e1adac607c3f436cb1, State: Initialized, Role: FOLLOWER
I20260810 18:13:11.006971 10665 consensus_queue.cc:260] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } }
I20260810 18:13:11.007406 10664 ts_tablet_manager.cc:1428] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
I20260810 18:13:11.007799 10637 heartbeater.cc:503] Master 127.23.65.126:40041 was elected leader, sending a full tablet report...
I20260810 18:13:11.008353 10665 ts_tablet_manager.cc:1428] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1: Time spent starting tablet: real 0.010s user 0.000s sys 0.007s
I20260810 18:13:11.008772 10415 heartbeater.cc:503] Master 127.23.65.126:40041 was elected leader, sending a full tablet report...
I20260810 18:13:11.019284 10669 raft_consensus.cc:348] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } }
I20260810 18:13:11.019677 10669 raft_consensus.cc:374] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:13:11.019748 10669 raft_consensus.cc:729] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5980fa6c34394a9bac8e35454957333a, State: Initialized, Role: FOLLOWER
I20260810 18:13:11.020314 10669 consensus_queue.cc:260] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } }
I20260810 18:13:11.021240 10198 heartbeater.cc:503] Master 127.23.65.126:40041 was elected leader, sending a full tablet report...
I20260810 18:13:11.022766 10669 ts_tablet_manager.cc:1428] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a: Time spent starting tablet: real 0.020s user 0.011s sys 0.000s
W20260810 18:13:11.068852 10203 tablet.cc:2369] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:13:11.098614 10676 raft_consensus.cc:482] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:13:11.098852 10676 raft_consensus.cc:504] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } }
I20260810 18:13:11.102041 10676 leader_election.cc:288] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8612db7195764567b4d00afaa161fc44 (127.23.65.67:35881), 5980fa6c34394a9bac8e35454957333a (127.23.65.65:34035)
I20260810 18:13:11.108207 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:11.100963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:48131 (local address 127.23.65.67:35881)
0810 18:13:11.101099 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:11.101104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:11.102508 (+ 1404us) server_negotiation.cc:407] Connection header received
0810 18:13:11.102741 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:11.102745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:11.102800 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:11.102849 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:11.103495 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.104324 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.106584 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.107246 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.107350 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:11.107408 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:11.107961 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 18:13:11.108049 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:13:11.108716 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:11.100387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:35881 (local address 127.23.65.66:48131)
0810 18:13:11.102380 (+ 1993us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:11.102396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:11.102600 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:11.103004 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:11.103007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:11.103025 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:11.103353 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:11.103364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.104645 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.104649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:11.106307 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:11.106321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.107611 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.107615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:11.107799 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:11.107828 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:13:11.108437 (+ 609us) client_negotiation.cc:240] Negotiation successful
0810 18:13:11.108523 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1921,"thread_start_us":197,"threads_started":1}
I20260810 18:13:11.109681 10528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a446664d5d8a425abee3267f159f3931" candidate_uuid: "6cc54c386b5342e1adac607c3f436cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8612db7195764567b4d00afaa161fc44" is_pre_election: true
I20260810 18:13:11.110129 10528 raft_consensus.cc:2454] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc54c386b5342e1adac607c3f436cb1 in term 0.
I20260810 18:13:11.110801 10239 leader_election.cc:302] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc54c386b5342e1adac607c3f436cb1, 8612db7195764567b4d00afaa161fc44; no voters:
I20260810 18:13:11.111204 10676 raft_consensus.cc:2781] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:13:11.111307 10676 raft_consensus.cc:482] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:13:11.111358 10676 raft_consensus.cc:3037] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:13:11.112072 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:11.101893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:33179 (local address 127.23.65.65:34035)
0810 18:13:11.102003 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:11.102007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:11.105484 (+ 3477us) server_negotiation.cc:407] Connection header received
0810 18:13:11.105674 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:11.105677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:11.105724 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:11.105775 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:11.106403 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.107159 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.110278 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.110908 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.111013 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:11.111067 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:11.111857 (+ 790us) server_negotiation.cc:299] Negotiation successful
0810 18:13:11.111951 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:13:11.112722 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:11.101444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:34035 (local address 127.23.65.66:33179)
0810 18:13:11.105354 (+ 3910us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:11.105370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:11.105555 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:11.105929 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:11.105932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:11.105951 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:11.106227 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:11.106237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.107376 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.107379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:11.110093 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:11.110107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.111501 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.111505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:11.111689 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:11.111719 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:13:11.112413 (+ 694us) client_negotiation.cc:240] Negotiation successful
0810 18:13:11.112499 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3838,"thread_start_us":178,"threads_started":1}
I20260810 18:13:11.113437 10676 raft_consensus.cc:504] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } }
I20260810 18:13:11.113498 10113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a446664d5d8a425abee3267f159f3931" candidate_uuid: "6cc54c386b5342e1adac607c3f436cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5980fa6c34394a9bac8e35454957333a" is_pre_election: true
I20260810 18:13:11.113830 10676 leader_election.cc:288] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [CANDIDATE]: Term 1 election: Requested vote from peers 8612db7195764567b4d00afaa161fc44 (127.23.65.67:35881), 5980fa6c34394a9bac8e35454957333a (127.23.65.65:34035)
I20260810 18:13:11.113827 10113 raft_consensus.cc:2454] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc54c386b5342e1adac607c3f436cb1 in term 0.
I20260810 18:13:11.114981 10113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a446664d5d8a425abee3267f159f3931" candidate_uuid: "6cc54c386b5342e1adac607c3f436cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5980fa6c34394a9bac8e35454957333a"
I20260810 18:13:11.115099 10113 raft_consensus.cc:3037] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:13:11.115409 10528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a446664d5d8a425abee3267f159f3931" candidate_uuid: "6cc54c386b5342e1adac607c3f436cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8612db7195764567b4d00afaa161fc44"
I20260810 18:13:11.115564 10528 raft_consensus.cc:3037] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:13:11.116636 10113 raft_consensus.cc:2454] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc54c386b5342e1adac607c3f436cb1 in term 1.
I20260810 18:13:11.117139 10239 leader_election.cc:302] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5980fa6c34394a9bac8e35454957333a, 6cc54c386b5342e1adac607c3f436cb1; no voters:
I20260810 18:13:11.117460 10676 raft_consensus.cc:2781] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:13:11.117466 10528 raft_consensus.cc:2454] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc54c386b5342e1adac607c3f436cb1 in term 1.
I20260810 18:13:11.117808 10676 raft_consensus.cc:686] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 1 LEADER]: Becoming Leader. State: Replica: 6cc54c386b5342e1adac607c3f436cb1, State: Running, Role: LEADER
I20260810 18:13:11.118235 10676 consensus_queue.cc:237] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } }
I20260810 18:13:11.121965 9819 catalog_manager.cc:5495] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc54c386b5342e1adac607c3f436cb1 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "6cc54c386b5342e1adac607c3f436cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc54c386b5342e1adac607c3f436cb1" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 } health_report { overall_health: UNKNOWN } } }
W20260810 18:13:11.154906 10638 tablet.cc:2369] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:13:11.163415 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:11.158351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.23.65.66:38471)
0810 18:13:11.158489 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:11.158493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:11.158511 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:13:11.158602 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:11.158605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:11.158673 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:11.158785 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:13:11.159150 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.159956 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.161202 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:11.161523 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:11.161625 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:11.162049 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:13:11.162635 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:13:11.162758 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:11.163203 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 18:13:11.163267 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:13:11.170998 10113 raft_consensus.cc:1261] T a446664d5d8a425abee3267f159f3931 P 5980fa6c34394a9bac8e35454957333a [term 1 FOLLOWER]: Refusing update from remote peer 6cc54c386b5342e1adac607c3f436cb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:13:11.173120 10676 consensus_queue.cc:1035] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5980fa6c34394a9bac8e35454957333a" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:13:11.177822 10528 raft_consensus.cc:1261] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [term 1 FOLLOWER]: Refusing update from remote peer 6cc54c386b5342e1adac607c3f436cb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:13:11.178452 10676 consensus_queue.cc:1035] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8612db7195764567b4d00afaa161fc44" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 35881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:13:11.188243 10690 mvcc.cc:205] Tried to move back new op lower bound from 7317035381428715520 to 7317035381223792640. Current Snapshot: MvccSnapshot[applied={T|T < 7317035381428715520}]
I20260810 18:13:11.194444 10693 mvcc.cc:205] Tried to move back new op lower bound from 7317035381428715520 to 7317035381223792640. Current Snapshot: MvccSnapshot[applied={T|T < 7317035381428715520}]
I20260810 18:13:11.196312 9819 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:13:11.196480 9819 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:13:11.199206 9819 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 466fd9462dea4ca195daa14c0fac0534: Sending DeleteTablet for 3 replicas of tablet a446664d5d8a425abee3267f159f3931
I20260810 18:13:11.200316 10296 tablet_service.cc:1514] Processing DeleteTablet for tablet a446664d5d8a425abee3267f159f3931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:13:11 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:44488
I20260810 18:13:11.201048 10488 tablet_service.cc:1514] Processing DeleteTablet for tablet a446664d5d8a425abee3267f159f3931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:13:11 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:46094
I20260810 18:13:11.202316 10066 tablet_service.cc:1514] Processing DeleteTablet for tablet a446664d5d8a425abee3267f159f3931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:13:11 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:59450
I20260810 18:13:11.203708 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 9963
I20260810 18:13:11.220358 10707 tablet_replica.cc:330] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: stopping tablet replica
I20260810 18:13:11.220852 10706 tablet_replica.cc:330] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1: stopping tablet replica
I20260810 18:13:11.224265 10706 raft_consensus.cc:2229] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:13:11.224685 10707 raft_consensus.cc:2229] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:13:11.225150 10707 raft_consensus.cc:2258] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:13:11.225594 10706 raft_consensus.cc:2258] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 18:13:11.228387 9800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:34035 (error 108)
I20260810 18:13:11.229123 10707 ts_tablet_manager.cc:1905] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:13:11.229123 10706 ts_tablet_manager.cc:1905] T a446664d5d8a425abee3267f159f3931 P 6cc54c386b5342e1adac607c3f436cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:13:11.229509 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 10212
I20260810 18:13:11.235440 10707 ts_tablet_manager.cc:1918] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:13:11.235615 10707 log.cc:1196] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/a446664d5d8a425abee3267f159f3931
I20260810 18:13:11.236037 10707 ts_tablet_manager.cc:1939] T a446664d5d8a425abee3267f159f3931 P 8612db7195764567b4d00afaa161fc44: Deleting consensus metadata
I20260810 18:13:11.237797 9800 catalog_manager.cc:4841] TS 8612db7195764567b4d00afaa161fc44 (127.23.65.67:35881): tablet a446664d5d8a425abee3267f159f3931 (table test-table [id=f833db5ebf8b4ed387c5c2532f47f4cf]) successfully deleted
I20260810 18:13:11.239146 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:11.238711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:34035 (local address 127.0.0.1:59452)
0810 18:13:11.238958 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:11.239036 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260810 18:13:11.239336 9800 catalog_manager.cc:4584] TS 5980fa6c34394a9bac8e35454957333a (127.23.65.65:34035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a446664d5d8a425abee3267f159f3931: Network error: Client connection negotiation failed: client connection to 127.23.65.65:34035: connect: Connection refused (error 111)
Aug 10 18:13:11 dist-test-slave-jpj1 krb5kdc[8985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:11.249785 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:11.249412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:38471 (local address 127.0.0.1:44498)
0810 18:13:11.249622 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:11.249677 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.66:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260810 18:13:11.251039 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 10426
I20260810 18:13:11.263088 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 9770
I20260810 18:13:11.289103 23813 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 18:13:11.634594 23813 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 18:13:11.634739 23813 mini_ranger.cc:66] Stopping Ranger...
I20260810 18:13:11.978772 23813 mini_ranger.cc:68] Stopped Ranger
2026-08-10 18:13:11.979 UTC [9058] LOG: received smart shutdown request
2026-08-10 18:13:11.983 UTC [9058] LOG: background worker "logical replication launcher" (PID 9074) exited with exit code 1
2026-08-10 18:13:11.983 UTC [9065] LOG: shutting down
2026-08-10 18:13:12.136 UTC [9058] LOG: database system is shut down
2026-08-10T18:13:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48626 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:13:12 dist-test-slave-jpj1 krb5kdc[10730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:13:12 dist-test-slave-jpj1 krb5kdc[10730](info): set up 2 sockets
Aug 10 18:13:12 dist-test-slave-jpj1 krb5kdc[10730](info): commencing operation
krb5kdc: starting...
W20260810 18:13:13.387171 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:13:13 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:13:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:13:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.65.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.65.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/rangeradmin_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/rangeradmin_127.23.65.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.65.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.65.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/rangerlookup_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/rangerlookup_127.23.65.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.65.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.65.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/HTTP_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/HTTP_127.23.65.84@KRBTEST.COM.keytab.
I20260810 18:13:13.557811 23813 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmz0U2F/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/postgres -l logfile start
2026-08-10 18:13:15.864 UTC [10780] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 18:13:15.864 UTC [10780] LOG: listening on IPv4 address "127.23.65.84", port 60193
2026-08-10 18:13:15.868 UTC [10780] LOG: listening on Unix socket "/tmp/.s.PGSQL.60193"
2026-08-10 18:13:15.888 UTC [10783] LOG: database system was shut down at 2026-08-10 18:13:14 UTC
2026-08-10 18:13:15.892 UTC [10780] LOG: database system is ready to accept connections
I20260810 18:13:16.950657 23813 mini_postgres.cc:96] Postgres bound to 60193
2026-08-10 18:13:16.957 UTC [10795] FATAL: database "slave" does not exist
127.23.65.84:60193 - accepting connections
I20260810 18:13:16.958212 23813 mini_ranger.cc:161] Starting Ranger...
I20260810 18:13:16.973495 23813 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 18:13:17.412386 23813 mini_ranger.cc:87] Created ranger Postgres database
I20260810 18:13:17.412456 23813 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-admin
2026-08-10 18:13:17,699 [I] DB FLAVOR :POSTGRES
2026-08-10 18:13:17,700 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:13:17,700 [I] Checking connection..
2026-08-10 18:13:17,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 18:13:17,964 [I] Checking connection passed.
2026-08-10 18:13:17,964 [I] --------- Verifying version history table ---------
2026-08-10 18:13:17,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:13:18,226 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:13:18,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:13:18,477 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:13:18,477 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 18:13:18,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 18:13:18.725 UTC [10906] WARNING: there is no transaction in progress
2026-08-10 18:13:18,731 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 18:13:18,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:13:18,978 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 18:13:18,978 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:13:18,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 18:13:19,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 18:13:19,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jpj1.c.gcp-upstream.internal','N') ;"
2026-08-10 18:13:19,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:13:20,018 [I] Table x_portal_user does not exist in database ranger
2026-08-10 18:13:20,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:13:20,271 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 18:13:20,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:13:20,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:13:20,763 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 18:13:20,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 18:13:21.675 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.690 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.705 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.719 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.738 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.754 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.803 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.809 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.823 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.832 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.842 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.854 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.865 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.873 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.880 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:21.887 UTC [11092] WARNING: there is no transaction in progress
2026-08-10 18:13:22,499 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 18:13:22,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 18:13:22,761 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 18:13:22,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:13:23,026 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 18:13:23,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:13:23,290 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 18:13:23,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:13:23,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:13:23,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jpj1.c.gcp-upstream.internal';"
2026-08-10 18:13:24,047 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 18:13:24,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 18:13:24,279 [I] DB_PATCHES have already been applied
I20260810 18:13:24.285351 23813 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmz0U2F/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 18:13:24.285422 23813 mini_ranger.cc:193] Using host: 127.23.65.84
I20260810 18:13:24.289568 23813 mini_ranger.cc:227] Ranger admin URL: http://127.23.65.84:33271
Aug 10, 2026 6:13:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:13:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:13:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:13:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:13:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:13:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:13:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 18:13:25.518102 23813 mini_ranger.cc:160] Time spent starting Ranger: real 8.560s user 0.001s sys 0.010s
Aug 10, 2026 6:13:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:13:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 6:13:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:13:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:13:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 18:13:46.021884 23813 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.65.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.65.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/rangerkms_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.65.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/rangerkms_127.23.65.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.65.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.65.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.65.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/HTTP_127.23.65.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.65.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/HTTP_127.23.65.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 18:13:46 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385626, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 18:13:46.109376 23813 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 18:13:46.119426 23813 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 18:13:46.915771 23813 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 18:13:46.915858 23813 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-kms
I20260810 18:13:46.915876 23813 mini_ranger_kms.cc:227] Using postgres at 127.23.65.84:60193
2026-08-10 18:13:47,077 [I] DB FLAVOR :POSTGRES
2026-08-10 18:13:47,077 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:13:47,078 [I] Checking connection
2026-08-10 18:13:47,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 18:13:47,324 [I] connection success
2026-08-10 18:13:47,325 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 18:13:47,325 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 18:13:47,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 18:13:47,563 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 18:13:47,564 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:13:47,564 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 18:13:47,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.65.84:60193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 18:13:47,835 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 18:13:48.276980 23813 mini_ranger_kms.cc:310] Created kms service
I20260810 18:13:48.490787 23813 mini_ranger_kms.cc:326] Created kudu user
I20260810 18:13:48.535274 23813 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 18:13:48 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385626, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:48.914552 23813 mini_ranger_kms.cc:384] Added ranger policy
I20260810 18:13:48.914714 23813 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmz0U2F/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmz0U2F/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 18:13:48.914736 23813 mini_ranger_kms.cc:240] Using host: 127.23.65.84
I20260810 18:13:48.919415 23813 mini_ranger_kms.cc:276] Ranger KMS PID: 11392
I20260810 18:13:48.919502 23813 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.65.84:56089
Aug 10, 2026 6:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:13:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:13:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:13:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:13:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 18:13:51 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385631, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.65.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:51.432714 23813 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.323s user 0.029s sys 0.117s
I20260810 18:13:51.432845 23813 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 18:13:51.432916 23813 mini_ranger_kms.cc:401] 127.23.65.84:56089/kms/v1/keys
Aug 10 18:13:51 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385631, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.65.84@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
Aug 10 18:13:51 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385631, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.65.84@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
Aug 10 18:13:54 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:13:54.218299 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43687
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:43687
--ranger_config_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:56089/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:13:54.331758 11477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:54.332031 11477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:54.332070 11477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:54.336118 11477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:13:54.336171 11477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:54.336189 11477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:54.336205 11477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:13:54.336222 11477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:13:54.341682 11477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.65.84:56089/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:43687
--ranger_config_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:54.342710 11477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:54.343719 11477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:54.352152 11485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:54.352164 11483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:54.352253 11477 server_base.cc:1034] running on GCE node
W20260810 18:13:54.352164 11489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:54.352952 11477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:54.353957 11477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:54.355139 11477 hybrid_clock.cc:648] HybridClock initialized: now 1786385634355126 us; error 24 us; skew 500 ppm
Aug 10 18:13:54 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385634, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:54.357779 11477 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:13:54.358606 11477 webserver.cc:460] Webserver started at http://127.23.65.126:40655/ using document root <none> and password file <none>
I20260810 18:13:54.359112 11477 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:54.359169 11477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:13:54.359355 11477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:13:54 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385634, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:54.409341 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "b8d4b303ffa14fb0b4ecc35c1330187a"
format_stamp: "Formatted at 2026-08-10 18:13:54 on dist-test-slave-jpj1"
server_key: "ddc62535d73618203fd2429af1accba5"
server_key_iv: "63192fd7f51e909e5811a244df015c82"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:54.410166 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "b8d4b303ffa14fb0b4ecc35c1330187a"
format_stamp: "Formatted at 2026-08-10 18:13:54 on dist-test-slave-jpj1"
server_key: "ddc62535d73618203fd2429af1accba5"
server_key_iv: "63192fd7f51e909e5811a244df015c82"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:54.415490 11477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 18:13:54.432659 11500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:54.434229 11477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:13:54.434394 11477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "b8d4b303ffa14fb0b4ecc35c1330187a"
format_stamp: "Formatted at 2026-08-10 18:13:54 on dist-test-slave-jpj1"
server_key: "ddc62535d73618203fd2429af1accba5"
server_key_iv: "63192fd7f51e909e5811a244df015c82"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:54.434535 11477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:54.448891 11477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:54.451817 11477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:54.452003 11477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:54.459388 11477 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:43687
I20260810 18:13:54.459400 11582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:43687 every 8 connection(s)
I20260810 18:13:54.460417 11477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:13:54.463944 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:13:54.466699 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 11477
I20260810 18:13:54.466835 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:13:54.467111 23813 external_mini_cluster.cc:1428] Setting key f7ec0f1ffd1c320a15f868b0db86e18f
I20260810 18:13:54.471051 11586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Bootstrap starting.
I20260810 18:13:54.473865 11586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Neither blocks nor log segments found. Creating new log.
I20260810 18:13:54.474819 11586 log.cc:824] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:13:54 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:13:54.477102 11586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: No bootstrap required, opened a new log
I20260810 18:13:54.481074 11589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:54.468571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.23.65.126:43687)
0810 18:13:54.468979 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:54.468996 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:54.469032 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:13:54.469876 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:54.469896 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:54.470223 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:54.470536 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:54.471330 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:54.472353 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:54.473200 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:54.473666 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:54.473819 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:54.476385 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:54.476390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:54.476403 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:54.476430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:54.478421 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:54.478829 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:54.478835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:54.478839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:54.478945 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:54.479201 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:54.479204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:54.479206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:54.479625 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:54.479823 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:54.480081 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 18:13:54.480363 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260810 18:13:54.480985 11586 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } }
I20260810 18:13:54.481267 11586 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:13:54.481387 11586 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d4b303ffa14fb0b4ecc35c1330187a, State: Initialized, Role: FOLLOWER
I20260810 18:13:54.481896 11586 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } }
I20260810 18:13:54.483114 11586 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:13:54.483268 11586 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:13:54.483412 11586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:13:54.484508 11586 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } }
I20260810 18:13:54.484863 11586 leader_election.cc:302] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8d4b303ffa14fb0b4ecc35c1330187a; no voters:
I20260810 18:13:54.485199 11586 leader_election.cc:288] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:13:54.485298 11592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:13:54.485543 11592 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 1 LEADER]: Becoming Leader. State: Replica: b8d4b303ffa14fb0b4ecc35c1330187a, State: Running, Role: LEADER
I20260810 18:13:54.485827 11592 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } }
I20260810 18:13:54.486492 11586 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:13:54.486956 11592 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8d4b303ffa14fb0b4ecc35c1330187a. Latest consensus state: current_term: 1 leader_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } } }
I20260810 18:13:54.487066 11592 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: This master's current role is: LEADER
I20260810 18:13:54.487619 11594 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } } }
I20260810 18:13:54.487884 11594 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: This master's current role is: LEADER
I20260810 18:13:54.488160 11606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:13:54.492568 11606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:13:54.493019 11586 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 18:13:54.503788 11606 catalog_manager.cc:1347] Generated new cluster ID: 1dab7a53a33e48e38ed28ec11e81654e
I20260810 18:13:54.503896 11606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:13:54.529161 11606 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:13:54.530586 11606 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:13:54.538558 11606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Generated new TSK 0
I20260810 18:13:54.539618 11606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:13:55.260874 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:56089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43687
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:13:55.415617 11661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:55.416157 11661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:55.416290 11661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:55.422559 11661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:55.422642 11661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:55.422735 11661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:13:55.430766 11661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.65.84:56089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:55.432337 11661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:55.433816 11661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:55.445580 11674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:55.445791 11676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:55.446147 11661 server_base.cc:1034] running on GCE node
W20260810 18:13:55.445943 11679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:55.446593 11661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:55.447216 11661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:55.448383 11661 hybrid_clock.cc:648] HybridClock initialized: now 1786385635448333 us; error 68 us; skew 500 ppm
Aug 10 18:13:55 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:55.452081 11661 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:13:55.453279 11661 webserver.cc:460] Webserver started at http://127.23.65.65:36323/ using document root <none> and password file <none>
I20260810 18:13:55.454033 11661 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:55.454146 11661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:13:55.454478 11661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:13:55 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:55.475459 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "54736cb0ca1b439682a8ff7882c17819"
format_stamp: "Formatted at 2026-08-10 18:13:55 on dist-test-slave-jpj1"
server_key: "286aa94b004cb2d2c0af3a5e98c5781c"
server_key_iv: "0a35423c4535871248cc8fe51c1b1034"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:55.476269 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "54736cb0ca1b439682a8ff7882c17819"
format_stamp: "Formatted at 2026-08-10 18:13:55 on dist-test-slave-jpj1"
server_key: "286aa94b004cb2d2c0af3a5e98c5781c"
server_key_iv: "0a35423c4535871248cc8fe51c1b1034"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:55.484469 11661 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.001s
I20260810 18:13:55.507979 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:55.509876 11661 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.001s
I20260810 18:13:55.510104 11661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "54736cb0ca1b439682a8ff7882c17819"
format_stamp: "Formatted at 2026-08-10 18:13:55 on dist-test-slave-jpj1"
server_key: "286aa94b004cb2d2c0af3a5e98c5781c"
server_key_iv: "0a35423c4535871248cc8fe51c1b1034"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:55.510360 11661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:55.537376 11661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:55.541685 11661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:55.542024 11661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:55.542776 11661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:13:55.543952 11661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:13:55.544070 11661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:55.544191 11661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:13:55.544273 11661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:55.555119 11661 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:32905
I20260810 18:13:55.557291 11868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:32905 every 8 connection(s)
I20260810 18:13:55.558218 11661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:13:55.562515 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 11661
I20260810 18:13:55.562659 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:13:55.562961 23813 external_mini_cluster.cc:1428] Setting key 024083612a6698f8ea851074b2ef5236
Aug 10 18:13:55 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:13:55.576184 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:55.561543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:43849 (local address 127.23.65.126:43687)
0810 18:13:55.561854 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:55.561859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:55.562576 (+ 717us) server_negotiation.cc:407] Connection header received
0810 18:13:55.563854 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:55.563858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:55.563917 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:55.563981 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:55.565595 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:55.566468 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:55.568222 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:55.568597 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:55.568672 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:55.571744 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:55.571748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:55.571751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:55.571779 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:55.573654 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:55.574307 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:55.574311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:55.574313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:55.574404 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:55.574805 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:55.574808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:55.574810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:55.575034 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:55.575113 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:55.575853 (+ 740us) server_negotiation.cc:299] Negotiation successful
0810 18:13:55.575994 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
I20260810 18:13:55.577967 11873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:55.561949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43687 (local address 127.23.65.65:43849)
0810 18:13:55.562423 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:55.562477 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:55.563525 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:55.564397 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:55.564406 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:55.564770 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:13:55.565426 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:55.565445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:55.566626 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:55.566629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:55.568085 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:55.568096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:55.568890 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:55.568893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:55.569085 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:55.569789 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:13:55.569809 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:13:55.571505 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:13:55.573812 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:55.573818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:55.573838 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:55.574187 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:55.574505 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:55.574508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:55.574511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:55.574687 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:55.575180 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:13:55.575188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:13:55.575521 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 18:13:55.576442 (+ 921us) client_negotiation.cc:240] Negotiation successful
0810 18:13:55.576829 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
I20260810 18:13:55.581617 11869 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43687
I20260810 18:13:55.581916 11869 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:55.582561 11869 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
I20260810 18:13:55.584462 11526 ts_manager.cc:194] Registered new tserver with Master: 54736cb0ca1b439682a8ff7882c17819 (127.23.65.65:32905)
I20260810 18:13:55.586051 11526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:43849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:13:55.602430 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:56089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43687
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:13:55 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 18:13:55.766425 11876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:55.766896 11876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:55.767021 11876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:55.773013 11876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:55.773103 11876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:55.773204 11876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:13:55.781312 11876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.65.84:56089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:55.782836 11876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:55.784325 11876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:55.797452 11886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:55.798189 11876 server_base.cc:1034] running on GCE node
W20260810 18:13:55.798635 11884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:55.798801 11882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:55.799335 11876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:55.800182 11876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:55.801380 11876 hybrid_clock.cc:648] HybridClock initialized: now 1786385635801348 us; error 47 us; skew 500 ppm
Aug 10 18:13:55 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:55.805007 11876 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:13:55.806100 11876 webserver.cc:460] Webserver started at http://127.23.65.66:38251/ using document root <none> and password file <none>
I20260810 18:13:55.806792 11876 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:55.806883 11876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:13:55.807104 11876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:13:55 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:55.834429 11876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "04c6c9d4c64245bf803d05b5db7a7a11"
format_stamp: "Formatted at 2026-08-10 18:13:55 on dist-test-slave-jpj1"
server_key: "41f217e0f36b571fce090e0a2fbf78d6"
server_key_iv: "e246456848b42a44edbc27befd427826"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:55.835035 11876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "04c6c9d4c64245bf803d05b5db7a7a11"
format_stamp: "Formatted at 2026-08-10 18:13:55 on dist-test-slave-jpj1"
server_key: "41f217e0f36b571fce090e0a2fbf78d6"
server_key_iv: "e246456848b42a44edbc27befd427826"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:55.840571 11876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 18:13:55.873108 11902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:55.875037 11876 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260810 18:13:55.875224 11876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "04c6c9d4c64245bf803d05b5db7a7a11"
format_stamp: "Formatted at 2026-08-10 18:13:55 on dist-test-slave-jpj1"
server_key: "41f217e0f36b571fce090e0a2fbf78d6"
server_key_iv: "e246456848b42a44edbc27befd427826"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:55.875393 11876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:55.887861 11876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:55.890919 11876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:55.891104 11876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:55.891780 11876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:13:55.892786 11876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:13:55.892866 11876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:55.892942 11876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:13:55.892972 11876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:55.903339 11876 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:36997
I20260810 18:13:55.904696 11876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:13:55.913602 12083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:36997 every 8 connection(s)
I20260810 18:13:55.915917 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 11876
I20260810 18:13:55.916066 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:13:55.916396 23813 external_mini_cluster.cc:1428] Setting key 6bd83dcad9417d35e4232420059552fc
Aug 10 18:13:55 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:13:55.927660 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:55.908262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43687 (local address 127.23.65.66:37339)
0810 18:13:55.910270 (+ 2008us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:55.910331 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:55.911458 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:55.912381 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:55.912390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:55.912754 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:13:55.913450 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:55.913472 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:55.914663 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:55.914666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:55.916113 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:55.916125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:55.917059 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:55.917064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:55.917267 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:55.920357 (+ 3090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:13:55.920379 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:13:55.922059 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:13:55.924238 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:55.924245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:55.924262 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:55.924616 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:55.924939 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:55.924942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:55.924944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:55.925132 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:55.925642 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:13:55.925649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:13:55.925983 (+ 334us) client_negotiation.cc:769] Sending connection context
0810 18:13:55.926199 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 18:13:55.926541 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"thread_start_us":149,"threads_started":1}
I20260810 18:13:55.928372 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:55.909435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:37339 (local address 127.23.65.126:43687)
0810 18:13:55.909611 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:55.909617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:55.910448 (+ 831us) server_negotiation.cc:407] Connection header received
0810 18:13:55.911762 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:55.911765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:55.911850 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:55.911918 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:55.913638 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:55.914488 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:55.916306 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:55.916632 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:55.916779 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:55.922294 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:55.922299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:55.922302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:55.922332 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:55.924076 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:55.924737 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:55.924741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:55.924743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:55.924835 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:55.925259 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:55.925263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:55.925264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:55.925498 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:55.925586 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:55.928071 (+ 2485us) server_negotiation.cc:299] Negotiation successful
0810 18:13:55.928218 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:13:55.930083 12084 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43687
I20260810 18:13:55.930361 12084 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:55.930954 12084 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
I20260810 18:13:55.932226 11526 ts_manager.cc:194] Registered new tserver with Master: 04c6c9d4c64245bf803d05b5db7a7a11 (127.23.65.66:36997)
I20260810 18:13:55.932977 11526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:37339
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:13:55.983780 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:56089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43687
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:13:56 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
W20260810 18:13:56.214614 12095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:56.215122 12095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:56.215245 12095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:56.221648 12095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:56.221817 12095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:56.221968 12095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:13:56.229910 12095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.65.84:56089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:56.235800 12095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:56.237392 12095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:56.252602 12101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:56.252912 12102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:56.254422 12095 server_base.cc:1034] running on GCE node
W20260810 18:13:56.254220 12105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:56.255021 12095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:56.255813 12095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:56.256974 12095 hybrid_clock.cc:648] HybridClock initialized: now 1786385636256903 us; error 89 us; skew 500 ppm
Aug 10 18:13:56 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:56.260596 12095 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:13:56.261871 12095 webserver.cc:460] Webserver started at http://127.23.65.67:34289/ using document root <none> and password file <none>
I20260810 18:13:56.262658 12095 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:56.262806 12095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:13:56.263085 12095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:13:56 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:56.287389 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "22be4b54681a49cfb98689c86c73542e"
format_stamp: "Formatted at 2026-08-10 18:13:56 on dist-test-slave-jpj1"
server_key: "14b3f2f230c8643c8d0151a601673f89"
server_key_iv: "4dd728b11e6bc0ed60bc3c025c1e0f44"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:56.288396 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "22be4b54681a49cfb98689c86c73542e"
format_stamp: "Formatted at 2026-08-10 18:13:56 on dist-test-slave-jpj1"
server_key: "14b3f2f230c8643c8d0151a601673f89"
server_key_iv: "4dd728b11e6bc0ed60bc3c025c1e0f44"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:56.294098 12095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 18:13:56.316238 12114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:56.318138 12095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:13:56.318276 12095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "22be4b54681a49cfb98689c86c73542e"
format_stamp: "Formatted at 2026-08-10 18:13:56 on dist-test-slave-jpj1"
server_key: "14b3f2f230c8643c8d0151a601673f89"
server_key_iv: "4dd728b11e6bc0ed60bc3c025c1e0f44"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:56.318411 12095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:56.346240 12095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:56.350200 12095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:56.350472 12095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:56.351336 12095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:13:56.352782 12095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:13:56.352905 12095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 18:13:56.353022 12095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:13:56.353184 12095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:56.381757 12095 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:34475
I20260810 18:13:56.383222 12095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:13:56.383948 12297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:34475 every 8 connection(s)
I20260810 18:13:56.393299 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 12095
I20260810 18:13:56.393448 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:13:56.393770 23813 external_mini_cluster.cc:1428] Setting key 3e99d8d81ae24e16a72b7b8c2b4d15a3
Aug 10 18:13:56 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:13:56.404428 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:56.388053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:60017 (local address 127.23.65.126:43687)
0810 18:13:56.388203 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:56.388208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:56.389547 (+ 1339us) server_negotiation.cc:407] Connection header received
0810 18:13:56.390834 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:56.390839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:56.390900 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:56.390993 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:56.393065 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:56.393877 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:56.395700 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:56.396043 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:56.396124 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:56.400034 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:56.400038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:56.400041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:56.400071 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:56.401858 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:56.402493 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:56.402497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:56.402499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:56.402609 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:56.403040 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:56.403045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:56.403046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:56.403281 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:56.403367 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:56.404117 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 18:13:56.404251 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:13:56.406078 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:56.388578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43687 (local address 127.23.65.67:60017)
0810 18:13:56.389368 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:56.389435 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:56.390559 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:56.391312 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:56.391322 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:56.391966 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:13:56.392861 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:56.392887 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:56.394050 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:56.394053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:56.395513 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:56.395525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:56.396370 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:56.396375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:56.396665 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:56.397426 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:13:56.397448 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:13:56.399837 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:13:56.401991 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:56.401997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:56.402013 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:56.402371 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:56.402723 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:56.402727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:56.402729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:56.402917 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:56.403439 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:13:56.403446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:13:56.403817 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 18:13:56.404611 (+ 794us) client_negotiation.cc:240] Negotiation successful
0810 18:13:56.404951 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":291,"threads_started":1}
I20260810 18:13:56.407704 12298 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43687
I20260810 18:13:56.407964 12298 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:56.408525 12298 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
I20260810 18:13:56.409708 11526 ts_manager.cc:194] Registered new tserver with Master: 22be4b54681a49cfb98689c86c73542e (127.23.65.67:34475)
I20260810 18:13:56.410460 11526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:60017
I20260810 18:13:56.420007 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:13:56.420753 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 11661
I20260810 18:13:56.438668 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 11876
I20260810 18:13:56.478121 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 12095
I20260810 18:13:56.488217 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 11477
I20260810 18:13:56.504570 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43687
--webserver_interface=127.23.65.126
--webserver_port=40655
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:43687
--ranger_config_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:56089/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.65.84:56089/kms with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:13:56.617043 12305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:56.617316 12305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:56.617358 12305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:56.621353 12305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:13:56.621417 12305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:56.621436 12305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:56.621454 12305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:13:56.621484 12305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:13:56.626873 12305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.65.84:56089/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:43687
--ranger_config_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:56.627924 12305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:56.628949 12305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:56.637045 12313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:56.637050 12312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:56.637228 12305 server_base.cc:1034] running on GCE node
W20260810 18:13:56.637051 12315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:56.637863 12305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:56.638918 12305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:56.640067 12305 hybrid_clock.cc:648] HybridClock initialized: now 1786385636640051 us; error 27 us; skew 500 ppm
Aug 10 18:13:56 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:56.642769 12305 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:13:56.643632 12305 webserver.cc:460] Webserver started at http://127.23.65.126:40655/ using document root <none> and password file <none>
I20260810 18:13:56.644120 12305 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:56.644183 12305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:13:56 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:56.723294 12305 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:13:56.726161 12326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:56.727831 12305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:13:56.727972 12305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "b8d4b303ffa14fb0b4ecc35c1330187a"
format_stamp: "Formatted at 2026-08-10 18:13:54 on dist-test-slave-jpj1"
server_key: "ddc62535d73618203fd2429af1accba5"
server_key_iv: "63192fd7f51e909e5811a244df015c82"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:56.728659 12305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:56.770802 12305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:56.773583 12305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:56.773741 12305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:56.781075 12305 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:43687
I20260810 18:13:56.781096 12412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:43687 every 8 connection(s)
I20260810 18:13:56.782159 12305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:13:56.782889 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 12305
I20260810 18:13:56.783309 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:56089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:32905
--local_ip_for_outbound_sockets=127.23.65.65
--tserver_master_addrs=127.23.65.126:43687
--webserver_port=36323
--webserver_interface=127.23.65.65
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.65.84:56089/kms with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:13:56.786495 12414 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:13:56.788166 12414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Bootstrap starting.
I20260810 18:13:56.804183 12414 log.cc:824] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:13:56.810459 12414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:13:56.810832 12414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Bootstrap complete.
I20260810 18:13:56.814779 12414 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } }
I20260810 18:13:56.815176 12414 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d4b303ffa14fb0b4ecc35c1330187a, State: Initialized, Role: FOLLOWER
I20260810 18:13:56.815681 12414 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } }
I20260810 18:13:56.815810 12414 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:13:56.815873 12414 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:13:56.815964 12414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:13:56.817834 12414 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } }
I20260810 18:13:56.818147 12414 leader_election.cc:302] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8d4b303ffa14fb0b4ecc35c1330187a; no voters:
I20260810 18:13:56.818445 12414 leader_election.cc:288] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:13:56.818549 12423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:13:56.818742 12423 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [term 2 LEADER]: Becoming Leader. State: Replica: b8d4b303ffa14fb0b4ecc35c1330187a, State: Running, Role: LEADER
I20260810 18:13:56.819103 12423 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } }
I20260810 18:13:56.819736 12414 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:13:56.820506 12424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } } }
I20260810 18:13:56.820633 12424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: This master's current role is: LEADER
I20260810 18:13:56.821002 12425 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8d4b303ffa14fb0b4ecc35c1330187a. Latest consensus state: current_term: 2 leader_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d4b303ffa14fb0b4ecc35c1330187a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43687 } } }
I20260810 18:13:56.821139 12425 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a [sys.catalog]: This master's current role is: LEADER
I20260810 18:13:56.821496 12440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:13:56.821923 12414 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 18:13:56.827982 12440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:13:56.829128 12440 catalog_manager.cc:1259] Loaded cluster ID: 1dab7a53a33e48e38ed28ec11e81654e
I20260810 18:13:56.829255 12440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:13:56.832713 12440 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:13:56.833566 12440 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Loaded TSK: 0
I20260810 18:13:56.834836 12440 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:13:56.922880 12416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:56.923133 12416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:56.923194 12416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:56.927215 12416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:56.927281 12416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:56.927362 12416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:13:56.934644 12416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.65.84:56089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:32905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=36323
--tserver_master_addrs=127.23.65.126:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:56.936465 12416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:56.937821 12416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:56.950505 12474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:56.950973 12416 server_base.cc:1034] running on GCE node
W20260810 18:13:56.950508 12480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:56.951277 12475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:56.951923 12416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:56.952730 12416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:56.953933 12416 hybrid_clock.cc:648] HybridClock initialized: now 1786385636953908 us; error 43 us; skew 500 ppm
Aug 10 18:13:56 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:56.957720 12416 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:13:56.958797 12416 webserver.cc:460] Webserver started at http://127.23.65.65:36323/ using document root <none> and password file <none>
I20260810 18:13:56.959474 12416 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:56.959599 12416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:13:56 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:56.989292 12416 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260810 18:13:56.992985 12497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:56.994781 12416 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:13:56.994925 12416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "54736cb0ca1b439682a8ff7882c17819"
format_stamp: "Formatted at 2026-08-10 18:13:55 on dist-test-slave-jpj1"
server_key: "286aa94b004cb2d2c0af3a5e98c5781c"
server_key_iv: "0a35423c4535871248cc8fe51c1b1034"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:56.995735 12416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:57.042407 12416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:57.046518 12416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:57.046732 12416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:57.047529 12416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:13:57.048681 12416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:13:57.048779 12416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:57.048870 12416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:13:57.048928 12416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:57.059713 12416 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:32905
I20260810 18:13:57.061064 12687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:32905 every 8 connection(s)
I20260810 18:13:57.061405 12416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:13:57.066004 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 12416
I20260810 18:13:57.066557 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:56089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:36997
--local_ip_for_outbound_sockets=127.23.65.66
--tserver_master_addrs=127.23.65.126:43687
--webserver_port=38251
--webserver_interface=127.23.65.66
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.65.84:56089/kms with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:13:57 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:13:57.082595 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:57.063670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:34923 (local address 127.23.65.126:43687)
0810 18:13:57.064327 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:57.064353 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:57.064826 (+ 473us) server_negotiation.cc:407] Connection header received
0810 18:13:57.065941 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:57.065952 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:57.066312 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:57.066628 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:57.068060 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.069676 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.072116 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.072884 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.073040 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:57.076160 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:57.076167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:57.076188 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:57.076225 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:57.078372 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:57.079037 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:57.079044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:57.079050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:57.079216 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:57.079687 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:57.079691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:57.079693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:57.080128 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:57.080254 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:57.081026 (+ 772us) server_negotiation.cc:299] Negotiation successful
0810 18:13:57.081388 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":126,"threads_started":1}
I20260810 18:13:57.084563 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:57.063807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43687 (local address 127.23.65.65:34923)
0810 18:13:57.064621 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:57.064677 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:57.065504 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:57.066961 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:57.066970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:57.067248 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:13:57.067851 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:57.067877 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.069856 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.069861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:57.071938 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:57.071954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.073352 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.073357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:57.073556 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:57.074251 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:13:57.074275 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:13:57.075899 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:13:57.078513 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:57.078519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:57.078534 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:57.078891 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:57.079342 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:57.079345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:57.079347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:57.079569 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:57.080325 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:13:57.080332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:13:57.080648 (+ 316us) client_negotiation.cc:769] Sending connection context
0810 18:13:57.083310 (+ 2662us) client_negotiation.cc:240] Negotiation successful
0810 18:13:57.083660 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":130,"threads_started":1}
I20260810 18:13:57.087313 12688 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43687
I20260810 18:13:57.087621 12688 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:57.088222 12688 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
W20260810 18:13:57.091691 12688 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:43687 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 18:13:57.091845 12688 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:57.092052 12688 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
I20260810 18:13:57.093000 12688 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:57.093209 12688 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
W20260810 18:13:57.094101 12688 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:13:57.212146 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:57.212440 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:57.212502 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:57.217478 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:57.217558 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:57.217645 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:13:57.223397 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.65.84:56089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:36997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=38251
--tserver_master_addrs=127.23.65.126:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:57.224596 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:57.225793 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:57.240388 12704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:57.240388 12706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:57.240713 12710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:57.247782 12694 server_base.cc:1034] running on GCE node
I20260810 18:13:57.248536 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:57.249442 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:57.252759 12694 hybrid_clock.cc:648] HybridClock initialized: now 1786385637252726 us; error 37 us; skew 500 ppm
Aug 10 18:13:57 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:57.255781 12694 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:13:57.256914 12694 webserver.cc:460] Webserver started at http://127.23.65.66:38251/ using document root <none> and password file <none>
I20260810 18:13:57.257498 12694 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:57.257586 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:13:57 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:57.283813 12694 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:13:57.287186 12724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:57.288919 12694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 18:13:57.289074 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "04c6c9d4c64245bf803d05b5db7a7a11"
format_stamp: "Formatted at 2026-08-10 18:13:55 on dist-test-slave-jpj1"
server_key: "41f217e0f36b571fce090e0a2fbf78d6"
server_key_iv: "e246456848b42a44edbc27befd427826"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:57.289839 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:57.310570 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:57.314548 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:57.314822 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:57.315645 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:13:57.316779 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:13:57.316893 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:57.316993 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:13:57.317054 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:57.330000 12694 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:36997
I20260810 18:13:57.330134 12938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:36997 every 8 connection(s)
I20260810 18:13:57.331558 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:13:57.338024 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 12694
I20260810 18:13:57.338558 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.65.84:56089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:34475
--local_ip_for_outbound_sockets=127.23.65.67
--tserver_master_addrs=127.23.65.126:43687
--webserver_port=34289
--webserver_interface=127.23.65.67
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.65.84:56089/kms with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:13:57 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:13:57.353744 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:57.334578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:40701 (local address 127.23.65.126:43687)
0810 18:13:57.334737 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:57.334741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:57.335785 (+ 1044us) server_negotiation.cc:407] Connection header received
0810 18:13:57.336975 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:57.336979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:57.337040 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:57.337150 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:57.338612 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.339139 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.340708 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.341043 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.341133 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:57.349030 (+ 7897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:57.349034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:57.349037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:57.349075 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:57.350963 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:57.351685 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:57.351689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:57.351691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:57.351798 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:57.352265 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:57.352268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:57.352270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:57.352534 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:57.352641 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:57.353423 (+ 782us) server_negotiation.cc:299] Negotiation successful
0810 18:13:57.353570 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:13:57.354874 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:57.335039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43687 (local address 127.23.65.66:40701)
0810 18:13:57.335614 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:57.335673 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:57.336746 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:57.337332 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:57.337341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:57.337709 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:13:57.338421 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:57.338442 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.339272 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.339275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:57.340561 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:57.340573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.341345 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.341348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:57.341539 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:57.342234 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:13:57.342255 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:13:57.348781 (+ 6526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:13:57.351109 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:57.351116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:57.351144 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:57.351556 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:57.351901 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:57.351905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:57.351907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:57.352134 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:57.352717 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:13:57.352725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:13:57.353090 (+ 365us) client_negotiation.cc:769] Sending connection context
0810 18:13:57.353356 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 18:13:57.353739 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
I20260810 18:13:57.356685 12939 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43687
I20260810 18:13:57.356976 12939 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:57.357903 12939 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
W20260810 18:13:57.359719 12939 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:43687 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 18:13:57.359845 12939 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:57.360000 12939 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
I20260810 18:13:57.360776 12939 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:57.360970 12939 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
W20260810 18:13:57.362493 12939 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:13:57.489890 12950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:13:57.490165 12950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:13:57.490223 12950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:13:57.495039 12950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:13:57.495164 12950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:13:57.495249 12950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:13:57.502161 12950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.65.84:56089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:34475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=34289
--tserver_master_addrs=127.23.65.126:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:13:57.503715 12950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:13:57.504917 12950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:13:57.516769 12957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:13:57.516780 12958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:57.517372 12950 server_base.cc:1034] running on GCE node
W20260810 18:13:57.520282 12961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:13:57.520938 12950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:13:57.521824 12950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:13:57.523026 12950 hybrid_clock.cc:648] HybridClock initialized: now 1786385637522994 us; error 47 us; skew 500 ppm
Aug 10 18:13:57 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:57.526679 12950 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:13:57.527901 12950 webserver.cc:460] Webserver started at http://127.23.65.67:34289/ using document root <none> and password file <none>
I20260810 18:13:57.528637 12950 fs_manager.cc:359] Metadata directory not provided
I20260810 18:13:57.528760 12950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:13:57 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:57.551344 12950 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:13:57.554719 12977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:57.556429 12950 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:13:57.556635 12950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "22be4b54681a49cfb98689c86c73542e"
format_stamp: "Formatted at 2026-08-10 18:13:56 on dist-test-slave-jpj1"
server_key: "14b3f2f230c8643c8d0151a601673f89"
server_key_iv: "4dd728b11e6bc0ed60bc3c025c1e0f44"
server_key_version: "kuduclusterkey@0"
I20260810 18:13:57.557381 12950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:13:57.578990 12950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:13:57.582748 12950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:13:57.582968 12950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:13:57.583865 12950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:13:57.584951 12950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:13:57.585050 12950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:57.585139 12950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:13:57.585196 12950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:13:57.598119 12950 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:34475
I20260810 18:13:57.598166 13187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:34475 every 8 connection(s)
I20260810 18:13:57.599551 12950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:13:57.604144 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 12950
Aug 10 18:13:57 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:13:57.614220 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:57.601587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:45263 (local address 127.23.65.126:43687)
0810 18:13:57.601746 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:57.601751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:57.602610 (+ 859us) server_negotiation.cc:407] Connection header received
0810 18:13:57.603565 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:57.603569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:57.603625 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:57.603738 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:57.605073 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.605776 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.606929 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.607291 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.607400 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:57.610240 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:57.610243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:57.610245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:57.610267 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:57.611792 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:57.612419 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:57.612423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:57.612425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:57.612514 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:57.612940 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:57.612944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:57.612945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:57.613162 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:57.613255 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:57.613924 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 18:13:57.614050 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:13:57.615702 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:57.604868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.23.65.126:43687)
0810 18:13:57.605139 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:57.605147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:57.607857 (+ 2710us) server_negotiation.cc:407] Connection header received
0810 18:13:57.608162 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:57.608165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:57.608214 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:57.608310 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:57.609252 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.609783 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.610994 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.611259 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.611352 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:57.612631 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:57.612633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:57.612635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:57.612653 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:57.614003 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:57.614525 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:57.614527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:57.614529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:57.614603 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:57.614857 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:57.614860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:57.614862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:57.615020 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:57.615201 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:57.615381 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 18:13:57.615497 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260810 18:13:57.615878 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:57.601955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43687 (local address 127.23.65.67:45263)
0810 18:13:57.602440 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:57.602492 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:57.603338 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:57.603884 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:57.603893 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:57.604281 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:13:57.604899 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:57.604921 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.605945 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.605948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:57.606805 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:57.606815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:57.607453 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:57.607456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:57.607652 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:57.608202 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:13:57.608223 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:13:57.610091 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:13:57.611927 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:57.611933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:57.611947 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:57.612307 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:57.612607 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:13:57.612610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:13:57.612612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:13:57.612814 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:13:57.613339 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:13:57.613345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:13:57.613630 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 18:13:57.614439 (+ 809us) client_negotiation.cc:240] Negotiation successful
0810 18:13:57.614789 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260810 18:13:57.617175 13188 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43687
I20260810 18:13:57.617435 13188 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:57.618026 13188 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
I20260810 18:13:57.619361 12354 ts_manager.cc:194] Registered new tserver with Master: 22be4b54681a49cfb98689c86c73542e (127.23.65.67:34475)
I20260810 18:13:57.620565 12354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:45263
Aug 10 18:13:57 dist-test-slave-jpj1 krb5kdc[10730](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:13:58.094345 12688 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:58.094704 12688 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
I20260810 18:13:58.095475 12354 ts_manager.cc:194] Registered new tserver with Master: 54736cb0ca1b439682a8ff7882c17819 (127.23.65.65:32905)
I20260810 18:13:58.096326 12354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:34923
Aug 10 18:13:58 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
I20260810 18:13:58.362746 12939 heartbeater.cc:464] Registering TS with master...
I20260810 18:13:58.363044 12939 heartbeater.cc:511] Master 127.23.65.126:43687 requested a full tablet report, sending...
I20260810 18:13:58.363906 12354 ts_manager.cc:194] Registered new tserver with Master: 04c6c9d4c64245bf803d05b5db7a7a11 (127.23.65.66:36997)
I20260810 18:13:58.364699 12354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:40701
I20260810 18:13:58.373888 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:13:58.385857 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.375621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42012 (local address 127.23.65.126:43687)
0810 18:13:58.376195 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:58.376200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:58.376530 (+ 330us) server_negotiation.cc:407] Connection header received
0810 18:13:58.376571 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:58.376574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:58.376642 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:58.376720 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:13:58.377659 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.378466 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.379715 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.380076 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.380179 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.381648 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:13:58.381652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:13:58.381655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:13:58.381682 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:13:58.383495 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:58.384036 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:58.384040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:58.384042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:58.384142 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:13:58.384422 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:13:58.384425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:13:58.384427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:13:58.384712 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:13:58.384840 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:58.385491 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 18:13:58.385675 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":85,"threads_started":1}
I20260810 18:13:58.390470 12354 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:13:58.395514 12354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:13:58.423489 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.408156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:32905 (local address 127.0.0.1:42214)
0810 18:13:58.412750 (+ 4594us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:58.412763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:58.412841 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:58.414200 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:58.414204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:58.414222 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:58.414317 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.414323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.415817 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.415820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.419560 (+ 3740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.419568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.422919 (+ 3351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.422922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.423035 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.423079 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 18:13:58.423191 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 18:13:58.423327 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4531,"thread_start_us":87,"threads_started":1}
I20260810 18:13:58.425536 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.407078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:36997 (local address 127.0.0.1:39758)
0810 18:13:58.410950 (+ 3872us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:58.411000 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:58.411121 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:58.412306 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:58.412312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:58.412338 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:58.412486 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.412493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.420869 (+ 8376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.420872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.422806 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.422817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.425129 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.425133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.425234 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.425258 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:13:58.425323 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:13:58.425392 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3694,"thread_start_us":108,"threads_started":1}
I20260810 18:13:58.427021 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.408535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.23.65.65:32905)
0810 18:13:58.411394 (+ 2859us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:58.411400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:58.412928 (+ 1528us) server_negotiation.cc:407] Connection header received
0810 18:13:58.413008 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:58.413012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:58.413904 (+ 892us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:58.414045 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:58.414439 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.415694 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.419692 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.420519 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.420629 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.420785 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:58.426747 (+ 5962us) server_negotiation.cc:299] Negotiation successful
0810 18:13:58.426874 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2750,"thread_start_us":125,"threads_started":1}
I20260810 18:13:58.432001 12586 tablet_service.cc:1467] Processing CreateTablet for tablet 51a975e77ccf44a28079fa882e2c6145 (DEFAULT_TABLE table=test-table [id=426366e96f50426f8035bd6d2613e215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:13:58.432688 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.407213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.23.65.66:36997)
0810 18:13:58.411632 (+ 4419us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:58.411638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:58.411664 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:13:58.411739 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:58.411744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:58.411949 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:58.412104 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:58.417705 (+ 5601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.418886 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.423927 (+ 5041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.424814 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.424917 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.425064 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:58.432321 (+ 7257us) server_negotiation.cc:299] Negotiation successful
0810 18:13:58.432446 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4285,"thread_start_us":115,"threads_started":1}
I20260810 18:13:58.433373 12586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a975e77ccf44a28079fa882e2c6145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:13:58.435863 12785 tablet_service.cc:1467] Processing CreateTablet for tablet 51a975e77ccf44a28079fa882e2c6145 (DEFAULT_TABLE table=test-table [id=426366e96f50426f8035bd6d2613e215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:13:58.436968 12785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a975e77ccf44a28079fa882e2c6145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:13:58.449052 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.409218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:34475 (local address 127.0.0.1:34530)
0810 18:13:58.419108 (+ 9890us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:58.419121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:58.419196 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:58.426471 (+ 7275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:58.426474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:58.426492 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:58.426579 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.426584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.430941 (+ 4357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.430945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.432150 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.432163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.447685 (+ 15522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.447689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.447791 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.447817 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:13:58.447884 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 18:13:58.447969 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9823,"thread_start_us":82,"threads_started":1}
I20260810 18:13:58.449841 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.409625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.23.65.67:34475)
0810 18:13:58.425799 (+ 16174us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:58.425807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:58.425834 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:13:58.425916 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:58.425923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:58.426163 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:58.426317 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:58.429716 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.430802 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.446431 (+ 15629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.447318 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.447430 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.447609 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:58.449553 (+ 1944us) server_negotiation.cc:299] Negotiation successful
0810 18:13:58.449698 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16047,"thread_start_us":120,"threads_started":1}
I20260810 18:13:58.451206 13228 tablet_bootstrap.cc:492] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: Bootstrap starting.
I20260810 18:13:58.451908 13227 tablet_bootstrap.cc:492] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819: Bootstrap starting.
I20260810 18:13:58.453168 13055 tablet_service.cc:1467] Processing CreateTablet for tablet 51a975e77ccf44a28079fa882e2c6145 (DEFAULT_TABLE table=test-table [id=426366e96f50426f8035bd6d2613e215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:13:58.453716 13228 tablet_bootstrap.cc:654] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: Neither blocks nor log segments found. Creating new log.
I20260810 18:13:58.454267 13055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a975e77ccf44a28079fa882e2c6145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:13:58.454478 13227 tablet_bootstrap.cc:654] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819: Neither blocks nor log segments found. Creating new log.
I20260810 18:13:58.454540 13228 log.cc:824] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: Log is configured to *not* fsync() on all Append() calls
I20260810 18:13:58.455251 13227 log.cc:824] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819: Log is configured to *not* fsync() on all Append() calls
I20260810 18:13:58.456753 13228 tablet_bootstrap.cc:492] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: No bootstrap required, opened a new log
I20260810 18:13:58.456970 13228 ts_tablet_manager.cc:1397] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:13:58.461263 13228 raft_consensus.cc:348] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } }
I20260810 18:13:58.462865 13228 raft_consensus.cc:374] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:13:58.462985 13228 raft_consensus.cc:729] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c6c9d4c64245bf803d05b5db7a7a11, State: Initialized, Role: FOLLOWER
I20260810 18:13:58.463634 13228 consensus_queue.cc:260] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } }
I20260810 18:13:58.464802 13228 ts_tablet_manager.cc:1428] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:13:58.465123 13227 tablet_bootstrap.cc:492] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819: No bootstrap required, opened a new log
I20260810 18:13:58.465199 12939 heartbeater.cc:503] Master 127.23.65.126:43687 was elected leader, sending a full tablet report...
I20260810 18:13:58.465378 13227 ts_tablet_manager.cc:1397] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20260810 18:13:58.469595 13227 raft_consensus.cc:348] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } }
I20260810 18:13:58.470485 13232 tablet_bootstrap.cc:492] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: Bootstrap starting.
I20260810 18:13:58.471946 13227 raft_consensus.cc:374] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:13:58.472065 13227 raft_consensus.cc:729] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54736cb0ca1b439682a8ff7882c17819, State: Initialized, Role: FOLLOWER
I20260810 18:13:58.472599 13227 consensus_queue.cc:260] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } }
I20260810 18:13:58.473058 13232 tablet_bootstrap.cc:654] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: Neither blocks nor log segments found. Creating new log.
I20260810 18:13:58.473690 13227 ts_tablet_manager.cc:1428] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260810 18:13:58.474009 13232 log.cc:824] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:13:58.474099 12688 heartbeater.cc:503] Master 127.23.65.126:43687 was elected leader, sending a full tablet report...
I20260810 18:13:58.476490 13232 tablet_bootstrap.cc:492] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: No bootstrap required, opened a new log
I20260810 18:13:58.476744 13232 ts_tablet_manager.cc:1397] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:13:58.481433 13232 raft_consensus.cc:348] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } }
I20260810 18:13:58.481755 13232 raft_consensus.cc:374] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:13:58.481807 13232 raft_consensus.cc:729] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22be4b54681a49cfb98689c86c73542e, State: Initialized, Role: FOLLOWER
I20260810 18:13:58.482333 13232 consensus_queue.cc:260] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } }
I20260810 18:13:58.483062 13188 heartbeater.cc:503] Master 127.23.65.126:43687 was elected leader, sending a full tablet report...
I20260810 18:13:58.483414 13232 ts_tablet_manager.cc:1428] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260810 18:13:58.563608 12689 tablet.cc:2369] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 18:13:58 dist-test-slave-jpj1 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for HTTP/127.23.65.84@KRBTEST.COM
W20260810 18:13:58.584163 12942 tablet.cc:2369] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:13:58.589849 13237 raft_consensus.cc:482] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:13:58.590118 13237 raft_consensus.cc:504] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } }
I20260810 18:13:58.591427 13237 leader_election.cc:288] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c6c9d4c64245bf803d05b5db7a7a11 (127.23.65.66:36997), 54736cb0ca1b439682a8ff7882c17819 (127.23.65.65:32905)
I20260810 18:13:58.598084 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.592127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:52741 (local address 127.23.65.65:32905)
0810 18:13:58.592408 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:58.592413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:58.593013 (+ 600us) server_negotiation.cc:407] Connection header received
0810 18:13:58.593254 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:58.593257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:58.593304 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:58.593350 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:58.594110 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.594909 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.596684 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.597269 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.597375 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.597433 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:58.597853 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 18:13:58.597947 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260810 18:13:58.598632 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.591714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:32905 (local address 127.23.65.67:52741)
0810 18:13:58.592886 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:58.592902 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:58.593121 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:58.593497 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:58.593500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:58.593524 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:58.593969 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.593979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.595029 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.595033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.596526 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.596539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.597530 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.597533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.597688 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.597716 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:13:58.598311 (+ 595us) client_negotiation.cc:240] Negotiation successful
0810 18:13:58.598390 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":2,"spinlock_wait_cycles":5248,"thread_start_us":432,"threads_started":1}
I20260810 18:13:58.599881 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.591786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:55023 (local address 127.23.65.66:36997)
0810 18:13:58.591926 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:58.591931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:58.593014 (+ 1083us) server_negotiation.cc:407] Connection header received
0810 18:13:58.593254 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:58.593257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:58.593304 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:58.593351 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:13:58.596048 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.596757 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.598470 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.599061 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.599178 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.599223 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:58.599661 (+ 438us) server_negotiation.cc:299] Negotiation successful
0810 18:13:58.599744 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:13:58.600339 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.591725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:36997 (local address 127.23.65.67:55023)
0810 18:13:58.592885 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:58.592902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:13:58.593121 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:13:58.593497 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:13:58.593499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:13:58.593524 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:13:58.595842 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.595854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.596895 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.596899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.598308 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:13:58.598321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.599319 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.599322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:13:58.599494 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.599518 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:13:58.600031 (+ 513us) client_negotiation.cc:240] Negotiation successful
0810 18:13:58.600113 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1105,"mutex_wait_us":21,"spinlock_wait_cycles":17408,"thread_start_us":247,"threads_started":1}
W20260810 18:13:58.601425 13189 tablet.cc:2369] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:13:58.601522 12825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a975e77ccf44a28079fa882e2c6145" candidate_uuid: "22be4b54681a49cfb98689c86c73542e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" is_pre_election: true
I20260810 18:13:58.601888 12825 raft_consensus.cc:2454] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22be4b54681a49cfb98689c86c73542e in term 0.
I20260810 18:13:58.602460 12979 leader_election.cc:302] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c6c9d4c64245bf803d05b5db7a7a11, 22be4b54681a49cfb98689c86c73542e; no voters:
I20260810 18:13:58.602860 13237 raft_consensus.cc:2781] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:13:58.602959 13237 raft_consensus.cc:482] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:13:58.603003 13237 raft_consensus.cc:3037] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:13:58.603924 12613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a975e77ccf44a28079fa882e2c6145" candidate_uuid: "22be4b54681a49cfb98689c86c73542e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54736cb0ca1b439682a8ff7882c17819" is_pre_election: true
I20260810 18:13:58.604198 12613 raft_consensus.cc:2454] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22be4b54681a49cfb98689c86c73542e in term 0.
I20260810 18:13:58.605340 13237 raft_consensus.cc:504] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } }
I20260810 18:13:58.605660 13237 leader_election.cc:288] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [CANDIDATE]: Term 1 election: Requested vote from peers 04c6c9d4c64245bf803d05b5db7a7a11 (127.23.65.66:36997), 54736cb0ca1b439682a8ff7882c17819 (127.23.65.65:32905)
I20260810 18:13:58.606086 12613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a975e77ccf44a28079fa882e2c6145" candidate_uuid: "22be4b54681a49cfb98689c86c73542e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54736cb0ca1b439682a8ff7882c17819"
I20260810 18:13:58.606216 12825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a975e77ccf44a28079fa882e2c6145" candidate_uuid: "22be4b54681a49cfb98689c86c73542e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c6c9d4c64245bf803d05b5db7a7a11"
I20260810 18:13:58.606215 12613 raft_consensus.cc:3037] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:13:58.606323 12825 raft_consensus.cc:3037] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:13:58.607935 12825 raft_consensus.cc:2454] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22be4b54681a49cfb98689c86c73542e in term 1.
I20260810 18:13:58.607935 12613 raft_consensus.cc:2454] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22be4b54681a49cfb98689c86c73542e in term 1.
I20260810 18:13:58.608204 12979 leader_election.cc:302] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c6c9d4c64245bf803d05b5db7a7a11, 22be4b54681a49cfb98689c86c73542e; no voters:
I20260810 18:13:58.608520 13237 raft_consensus.cc:2781] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:13:58.608778 13237 raft_consensus.cc:686] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 1 LEADER]: Becoming Leader. State: Replica: 22be4b54681a49cfb98689c86c73542e, State: Running, Role: LEADER
I20260810 18:13:58.609138 13237 consensus_queue.cc:237] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } }
I20260810 18:13:58.612185 12354 catalog_manager.cc:5495] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22be4b54681a49cfb98689c86c73542e (127.23.65.67). New cstate: current_term: 1 leader_uuid: "22be4b54681a49cfb98689c86c73542e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22be4b54681a49cfb98689c86c73542e" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 34475 } health_report { overall_health: HEALTHY } } }
I20260810 18:13:58.657886 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.652848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.23.65.67:34475)
0810 18:13:58.652971 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:13:58.652976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:13:58.652988 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:13:58.653072 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:13:58.653075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:13:58.653139 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:13:58.653236 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:13:58.653728 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.654481 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.655658 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:13:58.655988 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:13:58.656084 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:13:58.656593 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:13:58.657175 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:13:58.657292 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:13:58.657665 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 18:13:58.657725 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:13:58.665521 12613 raft_consensus.cc:1261] T 51a975e77ccf44a28079fa882e2c6145 P 54736cb0ca1b439682a8ff7882c17819 [term 1 FOLLOWER]: Refusing update from remote peer 22be4b54681a49cfb98689c86c73542e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:13:58.665552 12825 raft_consensus.cc:1261] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [term 1 FOLLOWER]: Refusing update from remote peer 22be4b54681a49cfb98689c86c73542e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:13:58.666350 13237 consensus_queue.cc:1035] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c6c9d4c64245bf803d05b5db7a7a11" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:13:58.666970 13237 consensus_queue.cc:1035] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "54736cb0ca1b439682a8ff7882c17819" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:13:58.678587 13253 mvcc.cc:205] Tried to move back new op lower bound from 7317035575966277632 to 7317035575745777664. Current Snapshot: MvccSnapshot[applied={T|T < 7317035575966277632}]
I20260810 18:13:58.686542 13259 mvcc.cc:205] Tried to move back new op lower bound from 7317035575966277632 to 7317035575745777664. Current Snapshot: MvccSnapshot[applied={T|T < 7317035575966277632}]
I20260810 18:13:58.689515 13257 mvcc.cc:205] Tried to move back new op lower bound from 7317035575966277632 to 7317035575745777664. Current Snapshot: MvccSnapshot[applied={T|T < 7317035575966277632}]
I20260810 18:13:58.696882 12354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:13:58.697074 12354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:13:58.700773 12354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8d4b303ffa14fb0b4ecc35c1330187a: Sending DeleteTablet for 3 replicas of tablet 51a975e77ccf44a28079fa882e2c6145
I20260810 18:13:58.701800 12586 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a975e77ccf44a28079fa882e2c6145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:13:58 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:42214
I20260810 18:13:58.701889 12785 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a975e77ccf44a28079fa882e2c6145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:13:58 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:39758
I20260810 18:13:58.702718 13055 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a975e77ccf44a28079fa882e2c6145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:13:58 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:34530
I20260810 18:13:58.704068 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 12416
I20260810 18:13:58.723217 13274 tablet_replica.cc:330] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: stopping tablet replica
I20260810 18:13:58.723788 13274 raft_consensus.cc:2229] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:13:58.727672 13273 tablet_replica.cc:330] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: stopping tablet replica
I20260810 18:13:58.728073 13273 raft_consensus.cc:2229] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:13:58.728482 13273 raft_consensus.cc:2258] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:13:58.730129 13274 raft_consensus.cc:2258] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:13:58.733180 13274 ts_tablet_manager.cc:1905] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:13:58.733179 13273 ts_tablet_manager.cc:1905] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:13:58.736064 13273 ts_tablet_manager.cc:1918] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:13:58.736186 13273 log.cc:1196] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/51a975e77ccf44a28079fa882e2c6145
I20260810 18:13:58.736521 13273 ts_tablet_manager.cc:1939] T 51a975e77ccf44a28079fa882e2c6145 P 04c6c9d4c64245bf803d05b5db7a7a11: Deleting consensus metadata
I20260810 18:13:58.737932 12329 catalog_manager.cc:4841] TS 04c6c9d4c64245bf803d05b5db7a7a11 (127.23.65.66:36997): tablet 51a975e77ccf44a28079fa882e2c6145 (table test-table [id=426366e96f50426f8035bd6d2613e215]) successfully deleted
I20260810 18:13:58.738338 13274 ts_tablet_manager.cc:1918] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260810 18:13:58.742738 12330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:32905 (error 108)
I20260810 18:13:58.742960 13274 log.cc:1196] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/51a975e77ccf44a28079fa882e2c6145
I20260810 18:13:58.743427 13274 ts_tablet_manager.cc:1939] T 51a975e77ccf44a28079fa882e2c6145 P 22be4b54681a49cfb98689c86c73542e: Deleting consensus metadata
I20260810 18:13:58.744428 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 12694
I20260810 18:13:58.744481 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:13:58.744149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:32905 (local address 127.0.0.1:42222)
0810 18:13:58.744299 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 18:13:58.744384 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:32905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 18:13:58.744675 12330 catalog_manager.cc:4584] TS 54736cb0ca1b439682a8ff7882c17819 (127.23.65.65:32905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51a975e77ccf44a28079fa882e2c6145: Network error: Client connection negotiation failed: client connection to 127.23.65.65:32905: connect: Connection refused (error 111)
I20260810 18:13:58.748394 12329 catalog_manager.cc:4841] TS 22be4b54681a49cfb98689c86c73542e (127.23.65.67:34475): tablet 51a975e77ccf44a28079fa882e2c6145 (table test-table [id=426366e96f50426f8035bd6d2613e215]) successfully deleted
I20260810 18:13:58.759581 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 12950
I20260810 18:13:58.770910 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 12305
I20260810 18:13:58.782573 23813 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 18:13:59.126680 23813 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 18:13:59.126808 23813 mini_ranger.cc:66] Stopping Ranger...
I20260810 18:13:59.470669 23813 mini_ranger.cc:68] Stopped Ranger
2026-08-10 18:13:59.470 UTC [10780] LOG: received smart shutdown request
2026-08-10 18:13:59.475 UTC [10780] LOG: background worker "logical replication launcher" (PID 10793) exited with exit code 1
2026-08-10 18:13:59.475 UTC [10784] LOG: shutting down
2026-08-10 18:13:59.623 UTC [10780] LOG: database system is shut down
2026-08-10T18:13:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47492 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:13:59 dist-test-slave-jpj1 krb5kdc[13291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:13:59 dist-test-slave-jpj1 krb5kdc[13291](info): set up 2 sockets
Aug 10 18:13:59 dist-test-slave-jpj1 krb5kdc[13291](info): commencing operation
krb5kdc: starting...
W20260810 18:14:00.892697 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:14:00 dist-test-slave-jpj1 krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:14:01.016785 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:34107
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:34107,127.23.65.125:34309,127.23.65.124:33085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:01.127898 13314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:01.128152 13314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:01.128206 13314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:01.132174 13314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:14:01.132225 13314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:01.132251 13314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:01.132274 13314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:14:01.132303 13314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:14:01.137383 13314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:34107,127.23.65.125:34309,127.23.65.124:33085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:34107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:01.138370 13314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:01.139339 13314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:01.147433 13326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:01.147488 13330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:01.147488 13323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:01.147630 13314 server_base.cc:1034] running on GCE node
I20260810 18:14:01.148363 13314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:01.149422 13314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:01.150575 13314 hybrid_clock.cc:648] HybridClock initialized: now 1786385641150570 us; error 40 us; skew 500 ppm
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:01.156169 13314 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:14:01.157015 13314 webserver.cc:460] Webserver started at http://127.23.65.126:33703/ using document root <none> and password file <none>
I20260810 18:14:01.157529 13314 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:01.157590 13314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:01.157836 13314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:01.159847 13314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "f1c3e09263d94cb4b9ff5db1f975faef"
format_stamp: "Formatted at 2026-08-10 18:14:01 on dist-test-slave-jpj1"
server_key: "2521a38e95c0d4cf13c518f240def485"
server_key_iv: "9abfc6bcd548bc980839669dcd0281d9"
server_key_version: "encryptionkey@0"
I20260810 18:14:01.160369 13314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "f1c3e09263d94cb4b9ff5db1f975faef"
format_stamp: "Formatted at 2026-08-10 18:14:01 on dist-test-slave-jpj1"
server_key: "2521a38e95c0d4cf13c518f240def485"
server_key_iv: "9abfc6bcd548bc980839669dcd0281d9"
server_key_version: "encryptionkey@0"
I20260810 18:14:01.163975 13314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:14:01.166253 13346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:01.167484 13314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:01.167654 13314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "f1c3e09263d94cb4b9ff5db1f975faef"
format_stamp: "Formatted at 2026-08-10 18:14:01 on dist-test-slave-jpj1"
server_key: "2521a38e95c0d4cf13c518f240def485"
server_key_iv: "9abfc6bcd548bc980839669dcd0281d9"
server_key_version: "encryptionkey@0"
I20260810 18:14:01.167752 13314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:01.209088 13314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:01.212049 13314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:01.212229 13314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:01.218982 13314 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:34107
I20260810 18:14:01.219101 13443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:34107 every 8 connection(s)
I20260810 18:14:01.219961 13314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:14:01.222720 13445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:01.223609 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 13314
I20260810 18:14:01.223735 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:14:01.224015 23813 external_mini_cluster.cc:1428] Setting key 0f0b89a4bfeafee539ef32d86af4deaf
I20260810 18:14:01.226538 13445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.65.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.125@KRBTEST.COM" created.
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:01.241493 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.229968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.23.65.126:34107)
0810 18:14:01.230280 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.230303 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.230349 (+ 46us) server_negotiation.cc:407] Connection header received
0810 18:14:01.230715 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.230719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.230905 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.231065 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.231967 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.232862 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.233896 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.234263 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.234405 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.236833 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.236840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.236854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.236881 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.238470 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.239010 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.239014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.239015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.239114 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.239434 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.239436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.239437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.239835 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.239940 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.240395 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.240659 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260810 18:14:01.241498 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.228979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34107 (local address 127.0.0.1:47814)
0810 18:14:01.229475 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.229517 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:01.230437 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:01.231076 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:01.231081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:01.231561 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:01.231828 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.231843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.232973 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.232976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.233787 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.233797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.234401 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.234405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.234504 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.235051 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:01.235067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:01.236666 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:01.238580 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.238590 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.238601 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.238908 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.239208 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.239211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.239213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.239356 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.239947 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:01.239954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:01.240144 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 18:14:01.240350 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 18:14:01.240750 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":76,"thread_start_us":135,"threads_started":1}
I20260810 18:14:01.244149 13445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:14:01.244887 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.244595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:34309 (local address 127.0.0.1:40340)
0810 18:14:01.244725 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.244814 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:14:01.245126 13353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
I20260810 18:14:01.246011 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.245818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:34309 (local address 127.0.0.1:40344)
0810 18:14:01.245913 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.245958 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 18:14:01.246153 13445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.125:34309: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab.
Entry for principal kudu/127.23.65.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab.
I20260810 18:14:01.252837 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.125:34309
--webserver_interface=127.23.65.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab
--principal=kudu/127.23.65.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:34107,127.23.65.125:34309,127.23.65.124:33085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:14:01.295403 13445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } attempt: 1
I20260810 18:14:01.296362 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.295959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:34309 (local address 127.0.0.1:40352)
0810 18:14:01.296171 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.296256 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 18:14:01.297209 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.296946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:34309 (local address 127.0.0.1:40360)
0810 18:14:01.297067 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.297134 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 18:14:01.297341 13445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.125:34309: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
W20260810 18:14:01.365024 13456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:01.365260 13456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:01.365298 13456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 18:14:01.365504 13445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } attempt: 2
I20260810 18:14:01.366333 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.366003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:34309 (local address 127.0.0.1:40374)
0810 18:14:01.366171 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.366243 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 18:14:01.367060 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.366838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:34309 (local address 127.0.0.1:40376)
0810 18:14:01.366942 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.366996 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 18:14:01.367175 13445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.125:34309: Network error: Client connection negotiation failed: client connection to 127.23.65.125:34309: connect: Connection refused (error 111)
W20260810 18:14:01.369618 13456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:14:01.369678 13456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:01.369695 13456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:01.369710 13456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:14:01.369724 13456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:14:01.374883 13456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:34107,127.23.65.125:34309,127.23.65.124:33085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab
--principal=kudu/127.23.65.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.125:34309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.65.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:01.375962 13456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:01.376946 13456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:01.384913 13466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:01.384913 13471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:01.385022 13467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:01.385281 13456 server_base.cc:1034] running on GCE node
I20260810 18:14:01.385802 13456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:01.386845 13456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:01.388003 13456 hybrid_clock.cc:648] HybridClock initialized: now 1786385641387986 us; error 28 us; skew 500 ppm
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:01.390714 13456 init.cc:377] Logged in from keytab as kudu/127.23.65.125@KRBTEST.COM (short username kudu)
I20260810 18:14:01.391595 13456 webserver.cc:460] Webserver started at http://127.23.65.125:39411/ using document root <none> and password file <none>
I20260810 18:14:01.392115 13456 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:01.392180 13456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:01.392376 13456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:01.394348 13456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/data/instance:
uuid: "5e933858f7e4411fb8f3fed1a14bca59"
format_stamp: "Formatted at 2026-08-10 18:14:01 on dist-test-slave-jpj1"
server_key: "e48cb1c2267ae4f6488ca89d93529ac0"
server_key_iv: "13a96a1364c78fcf374488c460ff1ed7"
server_key_version: "encryptionkey@0"
I20260810 18:14:01.394888 13456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/wal/instance:
uuid: "5e933858f7e4411fb8f3fed1a14bca59"
format_stamp: "Formatted at 2026-08-10 18:14:01 on dist-test-slave-jpj1"
server_key: "e48cb1c2267ae4f6488ca89d93529ac0"
server_key_iv: "13a96a1364c78fcf374488c460ff1ed7"
server_key_version: "encryptionkey@0"
I20260810 18:14:01.398483 13456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:14:01.400820 13481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:01.402268 13456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:14:01.402403 13456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/wal
uuid: "5e933858f7e4411fb8f3fed1a14bca59"
format_stamp: "Formatted at 2026-08-10 18:14:01 on dist-test-slave-jpj1"
server_key: "e48cb1c2267ae4f6488ca89d93529ac0"
server_key_iv: "13a96a1364c78fcf374488c460ff1ed7"
server_key_version: "encryptionkey@0"
I20260810 18:14:01.402518 13456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:01.420444 13456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:01.423476 13456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:01.423691 13456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:01.430696 13456 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.125:34309
I20260810 18:14:01.430701 13563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.125:34309 every 8 connection(s)
I20260810 18:14:01.431694 13456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
I20260810 18:14:01.434417 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:01.438097 13564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:14:01.439488 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 13456
I20260810 18:14:01.439632 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-1/wal/instance
I20260810 18:14:01.439934 23813 external_mini_cluster.cc:1428] Setting key cea69be80c50cedc62a682b7b978b0ea
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.124@KRBTEST.COM; defaulting to no policy
I20260810 18:14:01.451841 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.440012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.23.65.126:34107)
0810 18:14:01.440157 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.440162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.441049 (+ 887us) server_negotiation.cc:407] Connection header received
0810 18:14:01.442169 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.442173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.442237 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.442332 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.443459 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.444022 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.445674 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.445916 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.445996 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.448626 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.448630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.448633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.448665 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.449997 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.450466 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.450470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.450472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.450571 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.450838 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.450840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.450841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.451010 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.451081 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.451603 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.451712 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.124@KRBTEST.COM" created.
I20260810 18:14:01.452594 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.440370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34107 (local address 127.0.0.1:47826)
0810 18:14:01.440866 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.440924 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:01.441908 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:01.442465 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:01.442471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:01.442809 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:01.443295 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.443314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.444149 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.444152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.445542 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.445555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.446076 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.446081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.446295 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.446883 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:01.446904 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:01.448453 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:01.450095 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.450100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.450111 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.450365 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.450644 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.450645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.450647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.450772 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.451090 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:01.451095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:01.451337 (+ 242us) client_negotiation.cc:769] Sending connection context
0810 18:14:01.451487 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 18:14:01.451758 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260810 18:14:01.453965 13564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:14:01.458313 13445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } attempt: 3
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
I20260810 18:14:01.463050 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.454419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:34309 (local address 127.0.0.1:40386)
0810 18:14:01.454509 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.454522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:01.454584 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:01.455158 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:01.455161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:01.455319 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:01.455391 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.455396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.456326 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.456329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.456845 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.456852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.457387 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.457391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.457474 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.457859 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:01.457872 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:01.459242 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:01.461444 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.461447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.461450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.461676 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.461966 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.461969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.461971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.462056 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.462617 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:01.462620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:01.462703 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 18:14:01.462783 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 18:14:01.462883 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
I20260810 18:14:01.463665 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.454537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.23.65.125:34309)
0810 18:14:01.454855 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.454863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.454880 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:14:01.454908 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.454912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.455047 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.455159 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.455484 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.456213 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.456941 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.457290 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.457377 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.459387 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.459398 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.459412 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.459436 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.461305 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.461781 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.461787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.461789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.461885 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.462166 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.462169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.462171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.462494 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.462626 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.462900 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.463025 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":160,"threads_started":1}
I20260810 18:14:01.465466 13564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:14:01.466182 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.465919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34172)
0810 18:14:01.466029 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.466111 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 18:14:01.466389 13485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab.
Entry for principal kudu/127.23.65.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab.
I20260810 18:14:01.467177 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.466994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34186)
0810 18:14:01.467084 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.467129 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 18:14:01.467322 13564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:33085: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
I20260810 18:14:01.467401 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.458666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:34309 (local address 127.0.0.1:40396)
0810 18:14:01.458787 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.458801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:01.458875 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:01.459299 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:01.459303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:01.459447 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:01.459573 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.459580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.460434 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.460438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.461270 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.461282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.461862 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.461866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.461977 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.462348 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:01.462362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:01.463898 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:01.465838 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.465841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.465843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.466075 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.466381 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.466384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.466386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.466483 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.466920 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:01.466924 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:01.467027 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 18:14:01.467107 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 18:14:01.467249 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 18:14:01.467891 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.124:33085
--webserver_interface=127.23.65.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab
--principal=kudu/127.23.65.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:34107,127.23.65.125:34309,127.23.65.124:33085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:14:01.467973 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.458833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.23.65.125:34309)
0810 18:14:01.459087 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.459099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.459109 (+ 10us) server_negotiation.cc:407] Connection header received
0810 18:14:01.459134 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.459136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.459185 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.459256 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.459685 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.460337 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.461388 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.461707 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.461847 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.464035 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.464038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.464041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.464066 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.465731 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.466173 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.466176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.466177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.466288 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.466571 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.466574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.466576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.466781 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.467705 (+ 924us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.467775 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.467862 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
I20260810 18:14:01.468561 13445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:14:01.469204 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.468959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34200)
0810 18:14:01.469076 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.469132 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 18:14:01.469906 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.469671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34206)
0810 18:14:01.469807 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.469854 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260810 18:14:01.470052 13445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:33085: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
I20260810 18:14:01.501200 13445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } attempt: 1
I20260810 18:14:01.501963 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.501680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34214)
0810 18:14:01.501820 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.501883 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 18:14:01.502748 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.502523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34230)
0810 18:14:01.502627 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.502682 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 18:14:01.502902 13445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:33085: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
I20260810 18:14:01.516536 13564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } attempt: 1
I20260810 18:14:01.517287 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.517017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34242)
0810 18:14:01.517147 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.517202 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 18:14:01.517937 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.517719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34250)
0810 18:14:01.517836 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.517879 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 18:14:01.518074 13564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:33085: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
I20260810 18:14:01.578095 13445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } attempt: 2
I20260810 18:14:01.578877 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.578598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34258)
0810 18:14:01.578734 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.578793 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 18:14:01.579583 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.579354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34262)
0810 18:14:01.579449 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.579496 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 18:14:01.579698 13445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:33085: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
W20260810 18:14:01.579778 13576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:01.579998 13576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:01.580040 13576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:01.583906 13576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:14:01.583952 13576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:01.583969 13576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:01.583984 13576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:14:01.584003 13576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:14:01.586206 13564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } attempt: 2
I20260810 18:14:01.586854 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.586566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34274)
0810 18:14:01.586707 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.586772 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 18:14:01.587520 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.587325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34286)
0810 18:14:01.587414 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.587463 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 18:14:01.587653 13564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:33085: Network error: Client connection negotiation failed: client connection to 127.23.65.124:33085: connect: Connection refused (error 111)
I20260810 18:14:01.589246 13576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:34107,127.23.65.125:34309,127.23.65.124:33085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab
--principal=kudu/127.23.65.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.124:33085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.65.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:01.590191 13576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:01.591120 13576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:01.599124 13587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:01.599124 13582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:01.599192 13591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:01.599478 13576 server_base.cc:1034] running on GCE node
I20260810 18:14:01.600049 13576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:01.601069 13576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:01.602219 13576 hybrid_clock.cc:648] HybridClock initialized: now 1786385641602191 us; error 40 us; skew 500 ppm
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:01.604846 13576 init.cc:377] Logged in from keytab as kudu/127.23.65.124@KRBTEST.COM (short username kudu)
I20260810 18:14:01.605710 13576 webserver.cc:460] Webserver started at http://127.23.65.124:43787/ using document root <none> and password file <none>
I20260810 18:14:01.606215 13576 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:01.606295 13576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:01.606485 13576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:01.608537 13576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/data/instance:
uuid: "e65d04a790c24087a2479243197cd8f6"
format_stamp: "Formatted at 2026-08-10 18:14:01 on dist-test-slave-jpj1"
server_key: "c738312de5a1401cfec400762c55e725"
server_key_iv: "2670a09cf9a0606fe957e1961b7dc280"
server_key_version: "encryptionkey@0"
I20260810 18:14:01.609063 13576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/wal/instance:
uuid: "e65d04a790c24087a2479243197cd8f6"
format_stamp: "Formatted at 2026-08-10 18:14:01 on dist-test-slave-jpj1"
server_key: "c738312de5a1401cfec400762c55e725"
server_key_iv: "2670a09cf9a0606fe957e1961b7dc280"
server_key_version: "encryptionkey@0"
I20260810 18:14:01.612529 13576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:14:01.614831 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:01.615983 13576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:14:01.616104 13576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/wal
uuid: "e65d04a790c24087a2479243197cd8f6"
format_stamp: "Formatted at 2026-08-10 18:14:01 on dist-test-slave-jpj1"
server_key: "c738312de5a1401cfec400762c55e725"
server_key_iv: "2670a09cf9a0606fe957e1961b7dc280"
server_key_version: "encryptionkey@0"
I20260810 18:14:01.616202 13576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:01.629235 13576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:01.632128 13576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:01.632297 13576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:01.638670 13576 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.124:33085
I20260810 18:14:01.638728 13679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.124:33085 every 8 connection(s)
I20260810 18:14:01.639605 13576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
I20260810 18:14:01.642210 13680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:01.644119 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 13576
I20260810 18:14:01.644264 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385478759079-23813-0/minicluster-data/master-2/wal/instance
I20260810 18:14:01.644553 23813 external_mini_cluster.cc:1428] Setting key ed121b07cf8b6a36d4ee2a5c067fcd0f
I20260810 18:14:01.646433 13680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:01.657704 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.649363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.23.65.126:34107)
0810 18:14:01.649518 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.649523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.649539 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:14:01.649588 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.649590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.649629 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.649693 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.650442 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.650976 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.651881 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.652085 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.652168 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.654364 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.654368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.654371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.654398 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.655806 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.656283 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.656286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.656288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.656369 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.656686 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.656689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.656690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.656913 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.657012 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.657424 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.657542 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:14:01.658478 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.648832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:34107 (local address 127.0.0.1:47840)
0810 18:14:01.649226 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.649288 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:01.649390 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:01.649760 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:01.649765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:01.650173 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:01.650329 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.650335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.651097 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.651101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.651771 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.651782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.652218 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.652222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.652320 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.652800 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:01.652816 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:01.654222 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:01.655900 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.655910 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.655920 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.656199 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.656453 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.656456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.656458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.656601 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.657022 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:01.657027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:01.657163 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 18:14:01.657342 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 18:14:01.657650 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":149,"threads_started":1}
I20260810 18:14:01.658494 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.645847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.23.65.124:33085)
0810 18:14:01.646277 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.646297 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.646339 (+ 42us) server_negotiation.cc:407] Connection header received
0810 18:14:01.647202 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.647228 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.647679 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.648026 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.648806 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.650143 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.650968 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.651437 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.651610 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.653957 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.653964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.653977 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.653999 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.655709 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.656168 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.656172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.656173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.656313 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.656618 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.656620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.656621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.657072 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.657254 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.657482 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.657844 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":178,"server-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260810 18:14:01.659642 13680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:01.667351 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.660062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:34309 (local address 127.0.0.1:40402)
0810 18:14:01.660209 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.660221 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:01.660290 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:01.660578 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:01.660580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:01.660747 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:01.660838 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.660845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.661605 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.661609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.662094 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.662101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.662542 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.662546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.662625 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.663013 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:01.663023 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:01.664338 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:01.665972 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.665975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.665978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.666191 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.666458 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.666460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.666461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.666545 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.666905 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:01.666909 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:01.667004 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 18:14:01.667113 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 18:14:01.667228 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 18:14:01.667573 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.660230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.23.65.125:34309)
0810 18:14:01.660356 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.660360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.660375 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:14:01.660415 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.660418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.660466 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.660541 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.660986 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.661502 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.662205 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.662444 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.662531 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.664474 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.664477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.664479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.664504 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.665876 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.666288 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.666291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.666293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.666369 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.666632 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.666634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.666635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.666814 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.666887 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.667146 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.667261 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:14:01.668169 13680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:14:01.668812 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.660874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.23.65.126:34107)
0810 18:14:01.660968 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.660972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.661003 (+ 31us) server_negotiation.cc:407] Connection header received
0810 18:14:01.661121 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.661124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.661159 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.661221 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.661777 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.662220 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.662930 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.663144 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.663226 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.665694 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.665717 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.665720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.665767 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.667130 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.667578 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.667581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.667583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.667673 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.667953 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.667956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.667958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.668174 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.668394 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.668557 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.668671 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
I20260810 18:14:01.676247 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.668861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.23.65.124:33085)
0810 18:14:01.669142 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.669145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.669157 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:14:01.669183 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.669184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.669214 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.669271 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.669655 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.670305 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.671123 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.671345 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.671418 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.673288 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.673291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.673293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.673314 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.674632 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.675036 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.675039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.675040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.675121 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.675388 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.675391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.675393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.675627 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.675737 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.675974 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.676097 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":224}
I20260810 18:14:01.676265 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.668641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34296)
0810 18:14:01.669030 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.669042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:01.669110 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:01.669303 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:01.669305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:01.669455 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:01.669538 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.669544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.670411 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.670414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.671025 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.671035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.671442 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.671446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.671526 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.671938 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:01.671952 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:01.673164 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:01.674734 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.674736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.674738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.674934 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.675202 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.675206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.675207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.675304 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.675716 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:01.675719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:01.675815 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 18:14:01.675914 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 18:14:01.676042 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260810 18:14:01.678789 13564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } attempt: 3
I20260810 18:14:01.678849 13445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } attempt: 3
I20260810 18:14:01.679829 13680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6: Bootstrap starting.
I20260810 18:14:01.681187 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.669917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.23.65.125:34309)
0810 18:14:01.670011 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.670015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.670028 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:14:01.670074 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.670076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.670107 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.670162 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.670775 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.671387 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.672221 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.672504 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.672586 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.675072 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.675076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.675078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.675098 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.676547 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.677221 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.677224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.677225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.677299 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.679838 (+ 2539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.679841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.679843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.680066 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.680296 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.680901 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.681038 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 18:14:01.682777 13680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:01.683753 13680 log.cc:824] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
Aug 10 18:14:01 dist-test-slave-jpj1 krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
I20260810 18:14:01.685613 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:14:01.686015 13680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6: No bootstrap required, opened a new log
I20260810 18:14:01.688598 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.679288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.23.65.124:33085)
0810 18:14:01.679635 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.679645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.679657 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:14:01.679688 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.679719 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.679762 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.680310 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.680401 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.681093 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.682057 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.682379 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.682818 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.685022 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.685025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.685028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.685045 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.686448 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.686830 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.686834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.686835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.686928 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.687221 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.687224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.687226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.687409 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.687496 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.688259 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.688411 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260810 18:14:01.689210 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.679191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34314)
0810 18:14:01.679311 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.679326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:01.679405 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:01.679892 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:01.679895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:01.680062 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:01.680158 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.680165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.681219 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.681223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.681832 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.681843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.682510 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.682514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.682600 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.683010 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:01.683024 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:01.684353 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:01.686548 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.686551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.686553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.686743 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.687019 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.687022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.687024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.687128 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.687503 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:01.687506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:01.687625 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 18:14:01.687724 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 18:14:01.687857 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 18:14:01.690146 13680 raft_consensus.cc:348] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } }
I20260810 18:14:01.690421 13680 raft_consensus.cc:374] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:01.690496 13680 raft_consensus.cc:729] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65d04a790c24087a2479243197cd8f6, State: Initialized, Role: FOLLOWER
I20260810 18:14:01.690934 13680 consensus_queue.cc:260] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } }
I20260810 18:14:01.691016 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.679148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:33085 (local address 127.0.0.1:34304)
0810 18:14:01.679322 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:01.679335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:01.679397 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:01.679736 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:01.679739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:01.679915 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:01.680010 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.680017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.680985 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.680989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.681759 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:01.681771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.682609 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.682613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:01.682702 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.683153 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:01.683166 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:01.684855 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:01.689467 (+ 4612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.689470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.689472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.689725 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.690034 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:01.690037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:01.690039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:01.690142 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:01.690564 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:01.690567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:01.690655 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 18:14:01.690745 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:14:01.690868 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260810 18:14:01.691493 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.679216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.23.65.124:33085)
0810 18:14:01.679515 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.679519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.679551 (+ 32us) server_negotiation.cc:407] Connection header received
0810 18:14:01.679582 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.679584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.679619 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.680115 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.680192 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.680851 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.682153 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.682478 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.682895 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.685277 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.685284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.685287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.685314 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.689341 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.689857 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.689859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.689861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.689939 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.690261 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.690264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.690265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.690455 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.691180 (+ 725us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.691259 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.691359 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":243}
I20260810 18:14:01.692261 13445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef: Bootstrap starting.
I20260810 18:14:01.692337 13680 sys_catalog.cc:564] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:14:01.694446 13564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59: Bootstrap starting.
I20260810 18:14:01.694697 13445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:01.695595 13445 log.cc:824] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:01.697049 13564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:01.697875 13564 log.cc:824] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:01.694581 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.699083 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 18:14:01.698110 13726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:14:01.699222 13726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:14:01.700433 13564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59: No bootstrap required, opened a new log
I20260810 18:14:01.700133 13576 master_runner.cc:186] Error getting master registration for 127.23.65.126:34107: OK, instance_id { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" instance_seqno: 1786385641217139 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:14:01.703452 13576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:14:01.704849 13564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } }
I20260810 18:14:01.705171 13564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:01.705258 13564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e933858f7e4411fb8f3fed1a14bca59, State: Initialized, Role: FOLLOWER
I20260810 18:14:01.705904 13564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } }
I20260810 18:14:01.706247 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.687844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.23.65.124:33085)
0810 18:14:01.688736 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.688739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.695972 (+ 7233us) server_negotiation.cc:407] Connection header received
0810 18:14:01.696012 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.696015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.696067 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.696142 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.696999 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.697814 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.698857 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.699449 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.699524 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.701298 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.701302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.701305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.701330 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.703174 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.703799 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.703803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.703805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.703896 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.704485 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.704489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.704490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.705058 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.705422 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.705955 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.706077 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":103,"threads_started":1}
I20260810 18:14:01.707145 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.688191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.23.65.125:34309)
0810 18:14:01.688289 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.688293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.698424 (+ 10131us) server_negotiation.cc:407] Connection header received
0810 18:14:01.698464 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.698466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.698509 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.698576 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.700055 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.700756 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.701822 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.702140 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.702206 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.703827 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.703831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.703834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.703860 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.705358 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.705840 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.705844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.705846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.705931 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.706233 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.706236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.706238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.706442 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.706547 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.706892 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.707006 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:14:01.707490 13564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:14:01.708307 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:01.688529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.23.65.126:34107)
0810 18:14:01.688629 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:01.688633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:01.695168 (+ 6535us) server_negotiation.cc:407] Connection header received
0810 18:14:01.695338 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:01.695341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:01.695385 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:01.695431 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:01.696392 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.697090 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.698190 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:01.698501 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:01.698575 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:01.704437 (+ 5862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:01.704441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:01.704444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:01.704471 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:01.705996 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.706714 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.706718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.706720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.706811 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:01.707140 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:01.707144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:01.707145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:01.707368 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:01.707479 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:01.708034 (+ 555us) server_negotiation.cc:299] Negotiation successful
0810 18:14:01.708152 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:14:01.709699 13445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef: No bootstrap required, opened a new log
W20260810 18:14:01.713207 13749 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:14:01.713282 13749 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:14:01.713609 13445 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } }
I20260810 18:14:01.713831 13445 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:01.713887 13445 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1c3e09263d94cb4b9ff5db1f975faef, State: Initialized, Role: FOLLOWER
I20260810 18:14:01.711387 13456 master_runner.cc:186] Error getting master registration for 127.23.65.126:34107: OK, instance_id { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" instance_seqno: 1786385641217139 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:14:01.714388 13445 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } }
I20260810 18:14:01.714504 13456 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:14:01.715751 13445 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:14:01.721320 13770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:14:01.721457 13770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:14:01.723728 13733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.723873 13733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:14:01.724963 13314 master_runner.cc:263] No leader master found from master f1c3e09263d94cb4b9ff5db1f975faef
I20260810 18:14:01.725026 13314 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:14:01.727741 13751 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.727849 13751 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:14:01.752314 13751 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:14:01.752516 13751 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } }
I20260810 18:14:01.753556 13751 leader_election.cc:288] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e933858f7e4411fb8f3fed1a14bca59 (127.23.65.125:34309), e65d04a790c24087a2479243197cd8f6 (127.23.65.124:33085)
I20260810 18:14:01.754339 13523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e933858f7e4411fb8f3fed1a14bca59" is_pre_election: true
I20260810 18:14:01.754664 13523 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1c3e09263d94cb4b9ff5db1f975faef in term 0.
I20260810 18:14:01.755201 13353 leader_election.cc:302] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e933858f7e4411fb8f3fed1a14bca59, f1c3e09263d94cb4b9ff5db1f975faef; no voters:
I20260810 18:14:01.756299 13642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e65d04a790c24087a2479243197cd8f6" is_pre_election: true
I20260810 18:14:01.756590 13642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1c3e09263d94cb4b9ff5db1f975faef in term 0.
I20260810 18:14:01.757222 13751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:14:01.757342 13751 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:14:01.757385 13751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:01.758744 13751 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } }
I20260810 18:14:01.759078 13751 leader_election.cc:288] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [CANDIDATE]: Term 1 election: Requested vote from peers 5e933858f7e4411fb8f3fed1a14bca59 (127.23.65.125:34309), e65d04a790c24087a2479243197cd8f6 (127.23.65.124:33085)
I20260810 18:14:01.759593 13523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e933858f7e4411fb8f3fed1a14bca59"
I20260810 18:14:01.759711 13523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:01.760963 13523 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1c3e09263d94cb4b9ff5db1f975faef in term 1.
I20260810 18:14:01.761330 13353 leader_election.cc:302] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e933858f7e4411fb8f3fed1a14bca59, f1c3e09263d94cb4b9ff5db1f975faef; no voters:
I20260810 18:14:01.761785 13642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e65d04a790c24087a2479243197cd8f6"
I20260810 18:14:01.761920 13642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:01.763101 13642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1c3e09263d94cb4b9ff5db1f975faef in term 1.
I20260810 18:14:01.766218 13751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:14:01.766531 13751 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [term 1 LEADER]: Becoming Leader. State: Replica: f1c3e09263d94cb4b9ff5db1f975faef, State: Running, Role: LEADER
I20260810 18:14:01.766913 13751 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } }
I20260810 18:14:01.768878 13751 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1c3e09263d94cb4b9ff5db1f975faef. Latest consensus state: current_term: 1 leader_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.769064 13751 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:01.769490 13779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:14:01.770541 13779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:14:01.779969 13642 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [term 1 FOLLOWER]: Refusing update from remote peer f1c3e09263d94cb4b9ff5db1f975faef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:01.780750 13775 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:01.785181 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1c3e09263d94cb4b9ff5db1f975faef. Latest consensus state: current_term: 1 leader_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.785312 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:14:01.786716 13523 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [term 1 FOLLOWER]: Refusing update from remote peer f1c3e09263d94cb4b9ff5db1f975faef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:01.789743 13775 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:01.793864 13733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1c3e09263d94cb4b9ff5db1f975faef. Latest consensus state: current_term: 1 leader_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.793987 13733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:14:01.799758 13775 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.799906 13775 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:01.803180 13733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.803294 13733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:14:01.803900 13780 mvcc.cc:205] Tried to move back new op lower bound from 7317035588723220480 to 7317035588681093120. Current Snapshot: MvccSnapshot[applied={T|T < 7317035588723220480}]
I20260810 18:14:01.805932 13786 mvcc.cc:205] Tried to move back new op lower bound from 7317035588723220480 to 7317035588681093120. Current Snapshot: MvccSnapshot[applied={T|T < 7317035588723220480}]
I20260810 18:14:01.806703 13751 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.806813 13751 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:01.808032 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c3e09263d94cb4b9ff5db1f975faef" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 34107 } } peers { permanent_uuid: "5e933858f7e4411fb8f3fed1a14bca59" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 34309 } } peers { permanent_uuid: "e65d04a790c24087a2479243197cd8f6" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 33085 } } }
I20260810 18:14:01.808144 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:14:01.808943 13783 mvcc.cc:205] Tried to move back new op lower bound from 7317035588723220480 to 7317035588681093120. Current Snapshot: MvccSnapshot[applied={T|T < 7317035588723220480}]
I20260810 18:14:01.813993 13779 catalog_manager.cc:1347] Generated new cluster ID: f51f41a907d644aeb8ea9a832a8048c8
I20260810 18:14:01.814059 13779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:14:01.846501 13779 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:14:01.847959 13779 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:14:01.870829 13779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1c3e09263d94cb4b9ff5db1f975faef: Generated new TSK 0
I20260810 18:14:01.871861 13779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:14:02.700724 13726 catalog_manager.cc:1259] Loaded cluster ID: f51f41a907d644aeb8ea9a832a8048c8
I20260810 18:14:02.700814 13726 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6: loading cluster ID for follower catalog manager: success
I20260810 18:14:02.702493 13726 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6: acquiring CA information for follower catalog manager: success
I20260810 18:14:02.703675 13726 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e65d04a790c24087a2479243197cd8f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:14:02.714522 13749 catalog_manager.cc:1259] Loaded cluster ID: f51f41a907d644aeb8ea9a832a8048c8
I20260810 18:14:02.714597 13749 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59: loading cluster ID for follower catalog manager: success
I20260810 18:14:02.716321 13749 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59: acquiring CA information for follower catalog manager: success
I20260810 18:14:02.717523 13749 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e933858f7e4411fb8f3fed1a14bca59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:14:02.947067 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 13314
I20260810 18:14:02.962677 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 13456
I20260810 18:14:02.975920 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 13576
2026-08-10T18:14:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3253 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:14:03 dist-test-slave-jpj1 krb5kdc[13799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:14:03 dist-test-slave-jpj1 krb5kdc[13799](info): set up 2 sockets
Aug 10 18:14:03 dist-test-slave-jpj1 krb5kdc[13799](info): commencing operation
krb5kdc: starting...
W20260810 18:14:04.141676 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:14:04 dist-test-slave-jpj1 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:14:04.266628 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38259
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:04.377985 13820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:04.378232 13820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:04.378270 13820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:04.382447 13820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:14:04.382511 13820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:04.382530 13820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:04.382545 13820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:14:04.382561 13820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:14:04.387931 13820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:38259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:04.388939 13820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:04.389931 13820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:04.398399 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:04.398387 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:04.398387 13833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:04.398495 13820 server_base.cc:1034] running on GCE node
I20260810 18:14:04.399277 13820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:04.400403 13820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:04.401592 13820 hybrid_clock.cc:648] HybridClock initialized: now 1786385644401565 us; error 30 us; skew 500 ppm
Aug 10 18:14:04 dist-test-slave-jpj1 krb5kdc[13799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385644, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:04.404233 13820 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:14:04.405159 13820 webserver.cc:460] Webserver started at http://127.23.65.126:37155/ using document root <none> and password file <none>
I20260810 18:14:04.405694 13820 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:04.405758 13820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:04.405967 13820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:04.408031 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "43a74d16f52d45e49cfdb91715ad4c34"
format_stamp: "Formatted at 2026-08-10 18:14:04 on dist-test-slave-jpj1"
server_key: "b9315893ddbca1ef393c2900a64d4091"
server_key_iv: "796fd62d23280d7b560654280bd679da"
server_key_version: "encryptionkey@0"
I20260810 18:14:04.408618 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "43a74d16f52d45e49cfdb91715ad4c34"
format_stamp: "Formatted at 2026-08-10 18:14:04 on dist-test-slave-jpj1"
server_key: "b9315893ddbca1ef393c2900a64d4091"
server_key_iv: "796fd62d23280d7b560654280bd679da"
server_key_version: "encryptionkey@0"
I20260810 18:14:04.412315 13820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:14:04.414628 13852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:04.415853 13820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:14:04.415970 13820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "43a74d16f52d45e49cfdb91715ad4c34"
format_stamp: "Formatted at 2026-08-10 18:14:04 on dist-test-slave-jpj1"
server_key: "b9315893ddbca1ef393c2900a64d4091"
server_key_iv: "796fd62d23280d7b560654280bd679da"
server_key_version: "encryptionkey@0"
I20260810 18:14:04.416071 13820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:04.424592 13820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:04.427105 13820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:04.427268 13820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:04.434028 13820 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:38259
I20260810 18:14:04.434037 13942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:38259 every 8 connection(s)
I20260810 18:14:04.434971 13820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:14:04.437676 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:04.442885 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 13820
I20260810 18:14:04.443030 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:14:04.443156 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34: Bootstrap starting.
I20260810 18:14:04.443351 23813 external_mini_cluster.cc:1428] Setting key 931b72b9f7968bc51316032a8c676abb
I20260810 18:14:04.445643 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:04.446347 13944 log.cc:824] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:04.448179 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34: No bootstrap required, opened a new log
Aug 10 18:14:04 dist-test-slave-jpj1 krb5kdc[13799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:04.451800 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a74d16f52d45e49cfdb91715ad4c34" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38259 } }
I20260810 18:14:04.452020 13944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:04.452068 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a74d16f52d45e49cfdb91715ad4c34, State: Initialized, Role: FOLLOWER
I20260810 18:14:04.452481 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a74d16f52d45e49cfdb91715ad4c34" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38259 } }
I20260810 18:14:04.452620 13944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:14:04.452688 13944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:14:04.452790 13944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:04.453686 13944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a74d16f52d45e49cfdb91715ad4c34" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38259 } }
I20260810 18:14:04.453994 13944 leader_election.cc:302] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43a74d16f52d45e49cfdb91715ad4c34; no voters:
I20260810 18:14:04.454262 13944 leader_election.cc:288] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:14:04.454357 13951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:14:04.454631 13951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [term 1 LEADER]: Becoming Leader. State: Replica: 43a74d16f52d45e49cfdb91715ad4c34, State: Running, Role: LEADER
I20260810 18:14:04.454977 13951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a74d16f52d45e49cfdb91715ad4c34" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38259 } }
I20260810 18:14:04.455441 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:14:04.458216 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43a74d16f52d45e49cfdb91715ad4c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a74d16f52d45e49cfdb91715ad4c34" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38259 } } }
I20260810 18:14:04.458338 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:04.457629 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:04.444661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.23.65.126:38259)
0810 18:14:04.445076 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:04.445091 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:04.445127 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:14:04.445752 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:04.445783 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:04.446112 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:04.446424 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:04.447234 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:04.448289 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:04.449122 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:04.449602 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:04.449747 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:04.452362 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:04.452368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:04.452380 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:04.452407 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:04.453985 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:04.454478 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:04.454484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:04.454490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:04.454632 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:04.454954 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:04.454957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:04.454959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:04.455455 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:04.455708 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:04.456503 (+ 795us) server_negotiation.cc:299] Negotiation successful
0810 18:14:04.456745 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260810 18:14:04.458771 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43a74d16f52d45e49cfdb91715ad4c34. Latest consensus state: current_term: 1 leader_uuid: "43a74d16f52d45e49cfdb91715ad4c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a74d16f52d45e49cfdb91715ad4c34" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38259 } } }
I20260810 18:14:04.458853 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34 [sys.catalog]: This master's current role is: LEADER
W20260810 18:14:04.460431 13969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:14:04.460505 13969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:14:04.460613 13971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:14:04.461338 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:14:04.461396 13971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:14:04.466374 13971 catalog_manager.cc:1347] Generated new cluster ID: 6d1841c41c3b490a8bd5aa75beef2e0c
I20260810 18:14:04.466436 13971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:14:04.478505 13971 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:14:04.479368 13971 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:14:04.483975 13971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43a74d16f52d45e49cfdb91715ad4c34: Generated new TSK 0
I20260810 18:14:04.484582 13971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:14:04.517257 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 13820
2026-08-10T18:14:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1540 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:14:04 dist-test-slave-jpj1 krb5kdc[13981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:14:04 dist-test-slave-jpj1 krb5kdc[13981](info): set up 2 sockets
Aug 10 18:14:04 dist-test-slave-jpj1 krb5kdc[13981](info): commencing operation
krb5kdc: starting...
W20260810 18:14:05.672161 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:14:05 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:14:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:14:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:14:05.793932 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46443
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:05.902786 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:05.903101 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:05.903144 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:05.907014 14013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:14:05.907066 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:05.907083 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:05.907101 14013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:14:05.907117 14013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:14:05.912155 14013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:05.913112 14013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:05.914039 14013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:05.921959 14023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:05.922125 14013 server_base.cc:1034] running on GCE node
W20260810 18:14:05.921968 14022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:05.921959 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:05.922727 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:05.923789 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:05.924947 14013 hybrid_clock.cc:648] HybridClock initialized: now 1786385645924933 us; error 24 us; skew 500 ppm
Aug 10 18:14:05 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385645, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:05.927570 14013 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:14:05.928363 14013 webserver.cc:460] Webserver started at http://127.23.65.126:34211/ using document root <none> and password file <none>
I20260810 18:14:05.928846 14013 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:05.928902 14013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:05.929059 14013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:05.930968 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "62ec900a2cbc4383b9f34e48e4fd54c6"
format_stamp: "Formatted at 2026-08-10 18:14:05 on dist-test-slave-jpj1"
server_key: "a94b13788e963b21c5f78563257d404d"
server_key_iv: "06d529849d30ee99e8268dc75aa30652"
server_key_version: "encryptionkey@0"
I20260810 18:14:05.931476 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "62ec900a2cbc4383b9f34e48e4fd54c6"
format_stamp: "Formatted at 2026-08-10 18:14:05 on dist-test-slave-jpj1"
server_key: "a94b13788e963b21c5f78563257d404d"
server_key_iv: "06d529849d30ee99e8268dc75aa30652"
server_key_version: "encryptionkey@0"
I20260810 18:14:05.934919 14013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:14:05.937251 14036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:05.938390 14013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 18:14:05.938490 14013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "62ec900a2cbc4383b9f34e48e4fd54c6"
format_stamp: "Formatted at 2026-08-10 18:14:05 on dist-test-slave-jpj1"
server_key: "a94b13788e963b21c5f78563257d404d"
server_key_iv: "06d529849d30ee99e8268dc75aa30652"
server_key_version: "encryptionkey@0"
I20260810 18:14:05.938575 14013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:05.949295 14013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:05.952282 14013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:05.952430 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:05.958750 14013 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:46443
I20260810 18:14:05.959025 14120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:46443 every 8 connection(s)
I20260810 18:14:05.959777 14013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:14:05.962435 14123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:05.968021 14123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Bootstrap starting.
I20260810 18:14:05.969350 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 14013
I20260810 18:14:05.969473 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:14:05.969749 23813 external_mini_cluster.cc:1428] Setting key 83613952a4bc110befddaf490f576a67
I20260810 18:14:05.970624 14123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:05.971388 14123 log.cc:824] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:05.973477 14123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: No bootstrap required, opened a new log
I20260810 18:14:05.977538 14123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:05.977829 14123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:05.977926 14123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ec900a2cbc4383b9f34e48e4fd54c6, State: Initialized, Role: FOLLOWER
Aug 10 18:14:05 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:05.978367 14123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:05.978518 14123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:14:05.978579 14123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:14:05.978686 14123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:05.979661 14123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:05.980016 14123 leader_election.cc:302] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ec900a2cbc4383b9f34e48e4fd54c6; no voters:
I20260810 18:14:05.980317 14123 leader_election.cc:288] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:14:05.980394 14133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:14:05.980594 14133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 1 LEADER]: Becoming Leader. State: Replica: 62ec900a2cbc4383b9f34e48e4fd54c6, State: Running, Role: LEADER
I20260810 18:14:05.980947 14133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:05.981670 14123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:14:05.982735 14136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ec900a2cbc4383b9f34e48e4fd54c6. Latest consensus state: current_term: 1 leader_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } } }
I20260810 18:14:05.982880 14136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:05.982731 14135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } } }
I20260810 18:14:05.983215 14135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:05.983399 14145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:14:05.984429 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:05.970790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.23.65.126:46443)
0810 18:14:05.971242 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:05.971257 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:05.971294 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:14:05.971999 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:05.972019 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:05.972341 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:05.972615 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:05.973493 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:05.974512 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:05.975682 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:05.976153 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:05.976264 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:05.978964 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:05.978969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:05.978983 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:05.979009 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:05.980742 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:05.981252 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:05.981256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:05.981260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:05.981372 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:05.981633 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:05.981636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:05.981638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:05.982116 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:05.983011 (+ 895us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:05.983172 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 18:14:05.983477 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":153,"threads_started":1}
I20260810 18:14:05.986581 14145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:14:05.991833 14145 catalog_manager.cc:1347] Generated new cluster ID: a688a7d20a384a3faf3cd7514d5c775d
I20260810 18:14:05.991904 14145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:14:06.002400 14145 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:14:06.003655 14145 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:14:06.013823 14145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Generated new TSK 0
I20260810 18:14:06.014657 14145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:14:06.055119 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46443
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:06.202371 14169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:06.202622 14169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:06.202663 14169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:06.206518 14169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:06.206563 14169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:06.206624 14169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:14:06.211588 14169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:06.212499 14169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:06.213460 14169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:06.222821 14182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:06.222821 14184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:06.222846 14180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:06.223273 14169 server_base.cc:1034] running on GCE node
I20260810 18:14:06.223706 14169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:06.224336 14169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:06.225493 14169 hybrid_clock.cc:648] HybridClock initialized: now 1786385646225469 us; error 36 us; skew 500 ppm
Aug 10 18:14:06 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385646, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:06.231716 14169 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:14:06.232573 14169 webserver.cc:460] Webserver started at http://127.23.65.65:36323/ using document root <none> and password file <none>
I20260810 18:14:06.233105 14169 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:06.233163 14169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:06.233335 14169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:06.235314 14169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "6ad4e02f99294f948529880658da3f4b"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "b9d1a8533a3a39e978f09673bb612b7f"
server_key_iv: "d3f1f3ad07a819665d85a5c768cc8834"
server_key_version: "encryptionkey@0"
I20260810 18:14:06.235946 14169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "6ad4e02f99294f948529880658da3f4b"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "b9d1a8533a3a39e978f09673bb612b7f"
server_key_iv: "d3f1f3ad07a819665d85a5c768cc8834"
server_key_version: "encryptionkey@0"
I20260810 18:14:06.239575 14169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:14:06.241959 14195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:06.243257 14169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:06.243364 14169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "6ad4e02f99294f948529880658da3f4b"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "b9d1a8533a3a39e978f09673bb612b7f"
server_key_iv: "d3f1f3ad07a819665d85a5c768cc8834"
server_key_version: "encryptionkey@0"
I20260810 18:14:06.243470 14169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:06.258302 14169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:06.261104 14169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:06.261277 14169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:06.261871 14169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:06.262681 14169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:06.262727 14169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:06.262766 14169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:06.262804 14169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:06.271343 14169 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:35353
I20260810 18:14:06.271370 14389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:35353 every 8 connection(s)
I20260810 18:14:06.272261 14169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
Aug 10 18:14:06 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385646, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:06.282505 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 14169
I20260810 18:14:06.282655 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:14:06.282944 23813 external_mini_cluster.cc:1428] Setting key 93fb8279101013c352dabc59914b0155
I20260810 18:14:06.286459 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.274353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:58501 (local address 127.23.65.126:46443)
0810 18:14:06.274543 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:06.274548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:06.275780 (+ 1232us) server_negotiation.cc:407] Connection header received
0810 18:14:06.277123 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:06.277126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:06.277183 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:06.277268 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:06.278422 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.278980 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.280129 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.280364 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.280443 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.282939 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:06.282943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:06.282945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:06.282970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:06.284492 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:06.284964 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:06.284967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:06.284968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:06.285042 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:06.285427 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:06.285430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:06.285432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:06.285603 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:06.285688 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:06.286159 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 18:14:06.286294 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 18:14:06.287201 14392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.274814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:58501)
0810 18:14:06.275617 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:06.275672 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:06.276914 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:06.277430 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:06.277436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:06.277789 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:06.278258 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.278281 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.279080 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.279083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.280035 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.280045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.280480 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.280483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.280629 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.281097 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:06.281113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:06.282782 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:06.284598 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:06.284604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:06.284618 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:06.284869 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:06.285124 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:06.285127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:06.285171 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:06.285336 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:06.285702 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:06.285708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:06.285937 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 18:14:06.286097 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 18:14:06.286415 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":149,"threads_started":1}
I20260810 18:14:06.288439 14390 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46443
I20260810 18:14:06.288662 14390 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:06.289131 14390 heartbeater.cc:511] Master 127.23.65.126:46443 requested a full tablet report, sending...
I20260810 18:14:06.290805 14064 ts_manager.cc:194] Registered new tserver with Master: 6ad4e02f99294f948529880658da3f4b (127.23.65.65:35353)
I20260810 18:14:06.292394 14064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:58501
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:14:06.310688 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46443
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:06.420900 14397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:06.421142 14397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:06.421197 14397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:06.425084 14397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:06.425138 14397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:06.425215 14397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:14:06.430227 14397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:06.431207 14397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:06.432281 14397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:06.441529 14407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:06.441522 14403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:06.441672 14397 server_base.cc:1034] running on GCE node
W20260810 18:14:06.441522 14412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:06.442281 14397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:06.442906 14397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:06.444082 14397 hybrid_clock.cc:648] HybridClock initialized: now 1786385646444048 us; error 40 us; skew 500 ppm
Aug 10 18:14:06 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385646, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:06.446782 14397 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:14:06.447687 14397 webserver.cc:460] Webserver started at http://127.23.65.66:34665/ using document root <none> and password file <none>
I20260810 18:14:06.448230 14397 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:06.448292 14397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:06.448457 14397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:06.450439 14397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "df806df0b0064f1e97583f7a0d798e83"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "d1b62f54d339936e9fc87ea67e50e9e1"
server_key_iv: "e56a8db09d49c1147ccb0eee60df1030"
server_key_version: "encryptionkey@0"
I20260810 18:14:06.450937 14397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "df806df0b0064f1e97583f7a0d798e83"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "d1b62f54d339936e9fc87ea67e50e9e1"
server_key_iv: "e56a8db09d49c1147ccb0eee60df1030"
server_key_version: "encryptionkey@0"
I20260810 18:14:06.454411 14397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:14:06.456652 14425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:06.457790 14397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:14:06.457888 14397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "df806df0b0064f1e97583f7a0d798e83"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "d1b62f54d339936e9fc87ea67e50e9e1"
server_key_iv: "e56a8db09d49c1147ccb0eee60df1030"
server_key_version: "encryptionkey@0"
I20260810 18:14:06.457984 14397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:06.485647 14397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:06.488440 14397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:06.488605 14397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:06.489197 14397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:06.490023 14397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:06.490069 14397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:06.490113 14397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:06.490141 14397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:06.498441 14397 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:40565
I20260810 18:14:06.498468 14602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:40565 every 8 connection(s)
I20260810 18:14:06.499279 14397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:14:06.506484 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 14397
I20260810 18:14:06.506610 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:14:06.506886 23813 external_mini_cluster.cc:1428] Setting key fb9c057ef913b944b5e2548c547ac3cb
Aug 10 18:14:06 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385646, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:06.512334 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.500977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:47729 (local address 127.23.65.126:46443)
0810 18:14:06.501100 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:06.501104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:06.501919 (+ 815us) server_negotiation.cc:407] Connection header received
0810 18:14:06.503218 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:06.503222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:06.503294 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:06.503363 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:06.504517 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.505042 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.506139 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.506426 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.506537 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.508974 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:06.508978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:06.508981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:06.509006 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:06.510379 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:06.510930 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:06.510932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:06.510933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:06.511005 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:06.511341 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:06.511343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:06.511344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:06.511499 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:06.511605 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:06.512123 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 18:14:06.512217 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:14:06.513386 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.501334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:47729)
0810 18:14:06.501780 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:06.501823 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:06.503020 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:06.503567 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:06.503576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:06.503922 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:06.504373 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.504387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.505162 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.505165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.506017 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.506027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.506583 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.506588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.506773 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.507285 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:06.507300 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:06.508822 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:06.510499 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:06.510506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:06.510520 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:06.510845 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:06.511077 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:06.511080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:06.511082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:06.511252 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:06.511610 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:06.511616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:06.511905 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 18:14:06.512095 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 18:14:06.512428 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260810 18:14:06.514613 14603 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46443
I20260810 18:14:06.514845 14603 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:06.515288 14603 heartbeater.cc:511] Master 127.23.65.126:46443 requested a full tablet report, sending...
I20260810 18:14:06.516335 14064 ts_manager.cc:194] Registered new tserver with Master: df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565)
I20260810 18:14:06.516865 14064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:47729
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:14:06.534217 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46443
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:06.642354 14615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:06.642580 14615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:06.642623 14615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:06.646533 14615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:06.646597 14615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:06.646673 14615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:14:06.651917 14615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:06.652927 14615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:06.653903 14615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:06.663165 14623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:06.663147 14629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:06.663165 14624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:06.663410 14615 server_base.cc:1034] running on GCE node
I20260810 18:14:06.663836 14615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:06.664402 14615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:06.665547 14615 hybrid_clock.cc:648] HybridClock initialized: now 1786385646665528 us; error 30 us; skew 500 ppm
Aug 10 18:14:06 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385646, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:06.668129 14615 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:14:06.668954 14615 webserver.cc:460] Webserver started at http://127.23.65.67:33025/ using document root <none> and password file <none>
I20260810 18:14:06.669469 14615 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:06.669534 14615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:06.669726 14615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:06.671680 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "5049583e02a1442d8cb4f39caa5eb0f7"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "29da7f3c719a535aea08332280834cb9"
server_key_iv: "3b28ff2c30012a4235ed55458bbc1877"
server_key_version: "encryptionkey@0"
I20260810 18:14:06.672204 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "5049583e02a1442d8cb4f39caa5eb0f7"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "29da7f3c719a535aea08332280834cb9"
server_key_iv: "3b28ff2c30012a4235ed55458bbc1877"
server_key_version: "encryptionkey@0"
I20260810 18:14:06.675668 14615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:14:06.677882 14642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:06.679009 14615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:06.679116 14615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "5049583e02a1442d8cb4f39caa5eb0f7"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "29da7f3c719a535aea08332280834cb9"
server_key_iv: "3b28ff2c30012a4235ed55458bbc1877"
server_key_version: "encryptionkey@0"
I20260810 18:14:06.679204 14615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:06.700945 14615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:06.704475 14615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:06.704686 14615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:06.705305 14615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:06.706177 14615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:06.706228 14615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:06.706286 14615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:06.706311 14615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:06.715149 14615 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:45379
I20260810 18:14:06.715174 14825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:45379 every 8 connection(s)
I20260810 18:14:06.716059 14615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:14:06.719604 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 14615
I20260810 18:14:06.719723 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:14:06.719990 23813 external_mini_cluster.cc:1428] Setting key 03f055165bb07970c0221908aaa96693
Aug 10 18:14:06 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385646, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:06.728940 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.717883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:35793 (local address 127.23.65.126:46443)
0810 18:14:06.718042 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:06.718047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:06.718908 (+ 861us) server_negotiation.cc:407] Connection header received
0810 18:14:06.720449 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:06.720452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:06.720505 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:06.720570 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:06.721866 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.722441 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.723593 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.723822 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.723898 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.726165 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:06.726168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:06.726170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:06.726193 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:06.727275 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:06.727762 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:06.727765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:06.727766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:06.727841 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:06.728084 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:06.728085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:06.728086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:06.728238 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:06.728303 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:06.728716 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 18:14:06.728818 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:14:06.729729 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.718264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:35793)
0810 18:14:06.718740 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:06.718786 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:06.720236 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:06.720749 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:06.720758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:06.721153 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:06.721718 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.721732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.722572 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.722575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.723462 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.723472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.723942 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.723945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.724088 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.724624 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:06.724641 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:06.726027 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:06.727368 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:06.727372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:06.727384 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:06.727678 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:06.727905 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:06.727907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:06.727908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:06.728029 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:06.728304 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:06.728309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:06.728524 (+ 215us) client_negotiation.cc:769] Sending connection context
0810 18:14:06.728664 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 18:14:06.728920 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260810 18:14:06.730783 14826 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46443
I20260810 18:14:06.730993 14826 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:06.731441 14826 heartbeater.cc:511] Master 127.23.65.126:46443 requested a full tablet report, sending...
I20260810 18:14:06.732434 14064 ts_manager.cc:194] Registered new tserver with Master: 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
I20260810 18:14:06.733059 14064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:35793
I20260810 18:14:06.733567 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:14:06.742044 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.735397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.23.65.126:46443)
0810 18:14:06.735499 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:06.735503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:06.735693 (+ 190us) server_negotiation.cc:407] Connection header received
0810 18:14:06.735798 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:06.735801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:06.735842 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:06.735905 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:06.736581 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.737070 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.737870 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.738084 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.738169 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.739348 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:06.739352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:06.739355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:06.739382 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:06.740500 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:06.740879 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:06.740882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:06.740884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:06.740958 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:06.741197 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:06.741200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:06.741202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:06.741405 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:06.741526 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:06.741769 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 18:14:06.741883 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 18:14:06.744860 14063 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:14:06.747830 14063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:14:06.760815 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.755459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:40565 (local address 127.0.0.1:54804)
0810 18:14:06.755766 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:06.755810 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:06.755924 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:06.756544 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:06.756549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:06.756575 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:06.756717 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.756725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.758144 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.758147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.759054 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.759066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.760152 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.760155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.760338 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.760383 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 18:14:06.760491 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 18:14:06.760598 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":22,"thread_start_us":90,"threads_started":1}
I20260810 18:14:06.760845 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.755635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.23.65.66:40565)
0810 18:14:06.755971 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:06.755984 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:06.756008 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:14:06.756091 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:06.756097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:06.756311 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:06.756870 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:06.757053 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.758034 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.759190 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.760056 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.760158 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.760314 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:06.760539 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 18:14:06.760687 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":139,"threads_started":1}
I20260810 18:14:06.761778 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.756123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:45379 (local address 127.0.0.1:56952)
0810 18:14:06.756792 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:06.756806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:06.756894 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:06.757405 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:06.757408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:06.757424 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:06.757516 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.757522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.759133 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.759135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.760317 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.760328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.761332 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.761336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.761422 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.761444 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:14:06.761524 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 18:14:06.761589 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":41,"thread_start_us":104,"threads_started":1}
I20260810 18:14:06.762008 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.756446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.23.65.65:35353)
0810 18:14:06.757003 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:06.757019 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:06.757131 (+ 112us) server_negotiation.cc:407] Connection header received
0810 18:14:06.757249 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:06.757253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:06.757404 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:06.757549 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:06.757809 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.758908 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.760099 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.761304 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.761405 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.761535 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:06.761737 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 18:14:06.761825 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":118,"threads_started":1}
I20260810 18:14:06.762089 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.756540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.23.65.67:45379)
0810 18:14:06.756974 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:06.756986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:06.757008 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:14:06.757075 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:06.757080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:06.757258 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:06.757419 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:06.757643 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.758793 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.760430 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.761224 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.761327 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.761459 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:06.761863 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 18:14:06.761957 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":123,"threads_started":1}
I20260810 18:14:06.762876 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:06.756148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:35353 (local address 127.0.0.1:58894)
0810 18:14:06.757049 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:06.757060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:06.757141 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:06.757582 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:06.757585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:06.757604 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:06.757697 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.757703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.759030 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.759033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.759987 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:06.759997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:06.761530 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:06.761534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:06.761611 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:06.761631 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:14:06.761698 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 18:14:06.761753 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":105,"threads_started":1}
I20260810 18:14:06.764063 14495 tablet_service.cc:1467] Processing CreateTablet for tablet 64f9915f45844d80ad44882d82a8c205 (DEFAULT_TABLE table=test-table [id=443eb879e3de402d95156f04eda46d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:06.764712 14720 tablet_service.cc:1467] Processing CreateTablet for tablet 64f9915f45844d80ad44882d82a8c205 (DEFAULT_TABLE table=test-table [id=443eb879e3de402d95156f04eda46d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:06.765220 14495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f9915f45844d80ad44882d82a8c205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:06.765724 14720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f9915f45844d80ad44882d82a8c205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:06.765666 14275 tablet_service.cc:1467] Processing CreateTablet for tablet 64f9915f45844d80ad44882d82a8c205 (DEFAULT_TABLE table=test-table [id=443eb879e3de402d95156f04eda46d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:06.766604 14275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f9915f45844d80ad44882d82a8c205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:06.772217 14852 tablet_bootstrap.cc:492] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Bootstrap starting.
I20260810 18:14:06.772847 14853 tablet_bootstrap.cc:492] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: Bootstrap starting.
I20260810 18:14:06.775123 14852 tablet_bootstrap.cc:654] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:06.775642 14853 tablet_bootstrap.cc:654] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:06.775835 14855 tablet_bootstrap.cc:492] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: Bootstrap starting.
I20260810 18:14:06.776460 14853 log.cc:824] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:06.776467 14852 log.cc:824] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:06.778301 14855 tablet_bootstrap.cc:654] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:06.778388 14853 tablet_bootstrap.cc:492] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: No bootstrap required, opened a new log
I20260810 18:14:06.778569 14853 ts_tablet_manager.cc:1397] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:14:06.778592 14852 tablet_bootstrap.cc:492] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: No bootstrap required, opened a new log
I20260810 18:14:06.778776 14852 ts_tablet_manager.cc:1397] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 18:14:06.779165 14855 log.cc:824] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:06.781296 14855 tablet_bootstrap.cc:492] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: No bootstrap required, opened a new log
I20260810 18:14:06.781483 14855 ts_tablet_manager.cc:1397] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 18:14:06.781730 14853 raft_consensus.cc:348] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:06.781970 14853 raft_consensus.cc:374] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:06.782029 14853 raft_consensus.cc:729] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5049583e02a1442d8cb4f39caa5eb0f7, State: Initialized, Role: FOLLOWER
I20260810 18:14:06.782483 14853 consensus_queue.cc:260] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:06.782481 14852 raft_consensus.cc:348] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:06.782811 14852 raft_consensus.cc:374] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:06.782868 14852 raft_consensus.cc:729] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad4e02f99294f948529880658da3f4b, State: Initialized, Role: FOLLOWER
I20260810 18:14:06.783499 14853 ts_tablet_manager.cc:1428] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:14:06.783850 14826 heartbeater.cc:503] Master 127.23.65.126:46443 was elected leader, sending a full tablet report...
I20260810 18:14:06.784217 14852 consensus_queue.cc:260] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:06.785041 14390 heartbeater.cc:503] Master 127.23.65.126:46443 was elected leader, sending a full tablet report...
I20260810 18:14:06.785192 14852 ts_tablet_manager.cc:1428] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:14:06.785410 14855 raft_consensus.cc:348] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:06.785586 14855 raft_consensus.cc:374] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:06.785614 14855 raft_consensus.cc:729] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df806df0b0064f1e97583f7a0d798e83, State: Initialized, Role: FOLLOWER
I20260810 18:14:06.785940 14855 consensus_queue.cc:260] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:06.786526 14603 heartbeater.cc:503] Master 127.23.65.126:46443 was elected leader, sending a full tablet report...
I20260810 18:14:06.786818 14855 ts_tablet_manager.cc:1428] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260810 18:14:06.967006 14827 tablet.cc:2369] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:14:07.000471 14605 tablet.cc:2369] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:14:07.023847 14391 tablet.cc:2369] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:14:07.032331 14862 raft_consensus.cc:482] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:14:07.032559 14862 raft_consensus.cc:504] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:07.033782 14862 leader_election.cc:288] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad4e02f99294f948529880658da3f4b (127.23.65.65:35353), 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
I20260810 18:14:07.037637 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.034253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:46025 (local address 127.23.65.67:45379)
0810 18:14:07.034419 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:07.034423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:07.034627 (+ 204us) server_negotiation.cc:407] Connection header received
0810 18:14:07.034759 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:07.034761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:07.034807 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:07.034871 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:07.035126 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.035659 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.036683 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.037071 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.037177 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.037230 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:07.037438 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 18:14:07.037513 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 18:14:07.037639 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.034128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:34539 (local address 127.23.65.65:35353)
0810 18:14:07.034271 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:07.034276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:07.034563 (+ 287us) server_negotiation.cc:407] Connection header received
0810 18:14:07.034700 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:07.034703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:07.034753 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:07.034817 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:07.035085 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.035608 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.036678 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.037058 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.037164 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.037223 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:07.037430 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 18:14:07.037507 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:14:07.037652 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.034060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:45379 (local address 127.23.65.66:46025)
0810 18:14:07.034536 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:07.034548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:07.034653 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:07.034889 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:07.034891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:07.034907 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:07.035027 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.035032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.035763 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.035767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.036562 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.036572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.037177 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.037181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.037320 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.037345 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:14:07.037407 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:14:07.037504 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":54,"thread_start_us":102,"threads_started":1}
I20260810 18:14:07.037658 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.034031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:35353 (local address 127.23.65.66:34539)
0810 18:14:07.034405 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:07.034417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:07.034584 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:07.034839 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:07.034842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:07.034858 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:07.034975 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.034981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.035709 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.035712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.036552 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.036562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.037164 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.037168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.037261 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.037335 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 18:14:07.037399 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 18:14:07.037493 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":55,"thread_start_us":90,"threads_started":1}
I20260810 18:14:07.038385 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f9915f45844d80ad44882d82a8c205" candidate_uuid: "df806df0b0064f1e97583f7a0d798e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" is_pre_election: true
I20260810 18:14:07.038436 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f9915f45844d80ad44882d82a8c205" candidate_uuid: "df806df0b0064f1e97583f7a0d798e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad4e02f99294f948529880658da3f4b" is_pre_election: true
I20260810 18:14:07.038702 14753 raft_consensus.cc:2454] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df806df0b0064f1e97583f7a0d798e83 in term 0.
I20260810 18:14:07.038746 14308 raft_consensus.cc:2454] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df806df0b0064f1e97583f7a0d798e83 in term 0.
I20260810 18:14:07.039187 14427 leader_election.cc:302] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad4e02f99294f948529880658da3f4b, df806df0b0064f1e97583f7a0d798e83; no voters:
I20260810 18:14:07.039456 14862 raft_consensus.cc:2781] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:14:07.039573 14862 raft_consensus.cc:482] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:14:07.039626 14862 raft_consensus.cc:3037] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:07.040877 14862 raft_consensus.cc:504] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:07.041244 14862 leader_election.cc:288] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [CANDIDATE]: Term 1 election: Requested vote from peers 6ad4e02f99294f948529880658da3f4b (127.23.65.65:35353), 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
I20260810 18:14:07.041621 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f9915f45844d80ad44882d82a8c205" candidate_uuid: "df806df0b0064f1e97583f7a0d798e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad4e02f99294f948529880658da3f4b"
I20260810 18:14:07.041744 14308 raft_consensus.cc:3037] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:07.042795 14308 raft_consensus.cc:2454] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df806df0b0064f1e97583f7a0d798e83 in term 1.
I20260810 18:14:07.043025 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f9915f45844d80ad44882d82a8c205" candidate_uuid: "df806df0b0064f1e97583f7a0d798e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5049583e02a1442d8cb4f39caa5eb0f7"
I20260810 18:14:07.043071 14427 leader_election.cc:302] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad4e02f99294f948529880658da3f4b, df806df0b0064f1e97583f7a0d798e83; no voters:
I20260810 18:14:07.043156 14753 raft_consensus.cc:3037] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:07.043241 14862 raft_consensus.cc:2781] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:14:07.043460 14862 raft_consensus.cc:686] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 1 LEADER]: Becoming Leader. State: Replica: df806df0b0064f1e97583f7a0d798e83, State: Running, Role: LEADER
I20260810 18:14:07.043846 14862 consensus_queue.cc:237] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:07.044503 14753 raft_consensus.cc:2454] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df806df0b0064f1e97583f7a0d798e83 in term 1.
I20260810 18:14:07.047281 14062 catalog_manager.cc:5495] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 reported cstate change: term changed from 0 to 1, leader changed from <none> to df806df0b0064f1e97583f7a0d798e83 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "df806df0b0064f1e97583f7a0d798e83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } health_report { overall_health: UNKNOWN } } }
I20260810 18:14:07.120929 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.117515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.23.65.66:40565)
0810 18:14:07.117659 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:07.117664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:07.117677 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:14:07.117759 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:07.117761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:07.117818 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:07.117918 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:14:07.118168 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.118675 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.119498 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.119733 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.119820 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.120129 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:14:07.120553 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:14:07.120659 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:07.120732 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 18:14:07.120790 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:14:07.126731 14308 raft_consensus.cc:1261] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Refusing update from remote peer df806df0b0064f1e97583f7a0d798e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:07.126731 14753 raft_consensus.cc:1261] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 1 FOLLOWER]: Refusing update from remote peer df806df0b0064f1e97583f7a0d798e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:07.127439 14862 consensus_queue.cc:1035] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:07.127635 14865 consensus_queue.cc:1035] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:07.143163 14063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:14:07.143330 14063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:14:07.145643 14063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Sending DeleteTablet for 3 replicas of tablet 64f9915f45844d80ad44882d82a8c205
I20260810 18:14:07.146476 14720 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f9915f45844d80ad44882d82a8c205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:07 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:56952
I20260810 18:14:07.146471 14495 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f9915f45844d80ad44882d82a8c205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:07 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:54804
I20260810 18:14:07.146492 14275 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f9915f45844d80ad44882d82a8c205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:07 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:58894
I20260810 18:14:07.147002 14886 tablet_replica.cc:330] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: stopping tablet replica
I20260810 18:14:07.147158 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 14169
I20260810 18:14:07.165370 14890 tablet_replica.cc:330] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: stopping tablet replica
I20260810 18:14:07.165428 14886 raft_consensus.cc:2229] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:14:07.165748 14890 raft_consensus.cc:2229] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:14:07.165860 14886 raft_consensus.cc:2258] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:07.166127 14890 raft_consensus.cc:2258] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:07.167486 14890 ts_tablet_manager.cc:1905] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:07.167587 14886 ts_tablet_manager.cc:1905] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:07.169811 14890 ts_tablet_manager.cc:1918] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:07.169906 14890 log.cc:1196] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/64f9915f45844d80ad44882d82a8c205
I20260810 18:14:07.170197 14890 ts_tablet_manager.cc:1939] T 64f9915f45844d80ad44882d82a8c205 P df806df0b0064f1e97583f7a0d798e83: Deleting consensus metadata
I20260810 18:14:07.170305 14886 ts_tablet_manager.cc:1918] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:07.170405 14886 log.cc:1196] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/64f9915f45844d80ad44882d82a8c205
I20260810 18:14:07.170737 14886 ts_tablet_manager.cc:1939] T 64f9915f45844d80ad44882d82a8c205 P 5049583e02a1442d8cb4f39caa5eb0f7: Deleting consensus metadata
I20260810 18:14:07.171453 14039 catalog_manager.cc:4841] TS df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565): tablet 64f9915f45844d80ad44882d82a8c205 (table test-table [id=443eb879e3de402d95156f04eda46d2f]) successfully deleted
I20260810 18:14:07.171890 14043 catalog_manager.cc:4841] TS 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379): tablet 64f9915f45844d80ad44882d82a8c205 (table test-table [id=443eb879e3de402d95156f04eda46d2f]) successfully deleted
W20260810 18:14:07.176460 14039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:35353 (error 108)
I20260810 18:14:07.176509 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 14397
I20260810 18:14:07.184826 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.184367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:35353 (local address 127.0.0.1:58900)
0810 18:14:07.184614 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:07.184713 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:35353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260810 18:14:07.185040 14039 catalog_manager.cc:4584] TS 6ad4e02f99294f948529880658da3f4b (127.23.65.65:35353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64f9915f45844d80ad44882d82a8c205: Network error: Client connection negotiation failed: client connection to 127.23.65.65:35353: connect: Connection refused (error 111)
I20260810 18:14:07.195427 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 14615
I20260810 18:14:07.207594 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 14013
I20260810 18:14:07.223724 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46443
--webserver_interface=127.23.65.126
--webserver_port=34211
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:07.338066 14896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:07.338335 14896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:07.338393 14896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:07.342356 14896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:14:07.342428 14896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:07.342455 14896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:07.342475 14896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:14:07.342496 14896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:14:07.347769 14896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:07.348821 14896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:07.349895 14896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:07.363945 14906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:07.364024 14896 server_base.cc:1034] running on GCE node
W20260810 18:14:07.363945 14911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:07.363998 14909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:07.364828 14896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:07.365940 14896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:07.367116 14896 hybrid_clock.cc:648] HybridClock initialized: now 1786385647367095 us; error 31 us; skew 500 ppm
Aug 10 18:14:07 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:07.369833 14896 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:14:07.370680 14896 webserver.cc:460] Webserver started at http://127.23.65.126:34211/ using document root <none> and password file <none>
I20260810 18:14:07.371236 14896 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:07.371315 14896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:07.375134 14896 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260810 18:14:07.377333 14918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:07.378569 14896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:07.378690 14896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "62ec900a2cbc4383b9f34e48e4fd54c6"
format_stamp: "Formatted at 2026-08-10 18:14:05 on dist-test-slave-jpj1"
server_key: "a94b13788e963b21c5f78563257d404d"
server_key_iv: "06d529849d30ee99e8268dc75aa30652"
server_key_version: "encryptionkey@0"
I20260810 18:14:07.379300 14896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:07.410790 14896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:07.414112 14896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:07.414314 14896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:07.422293 14896 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:46443
I20260810 18:14:07.422636 15004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:46443 every 8 connection(s)
I20260810 18:14:07.423509 14896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:14:07.428117 15007 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:14:07.429870 15007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Bootstrap starting.
I20260810 18:14:07.430986 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 14896
I20260810 18:14:07.431576 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:35353
--local_ip_for_outbound_sockets=127.23.65.65
--tserver_master_addrs=127.23.65.126:46443
--webserver_port=36323
--webserver_interface=127.23.65.65
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:14:07.448679 15007 log.cc:824] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:07.457818 15007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:14:07.458197 15007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Bootstrap complete.
I20260810 18:14:07.462328 15007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:07.462756 15007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ec900a2cbc4383b9f34e48e4fd54c6, State: Initialized, Role: FOLLOWER
I20260810 18:14:07.463267 15007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:07.463387 15007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:14:07.463456 15007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:14:07.463578 15007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:14:07.465569 15007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:07.465904 15007 leader_election.cc:302] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ec900a2cbc4383b9f34e48e4fd54c6; no voters:
I20260810 18:14:07.466212 15007 leader_election.cc:288] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:14:07.466322 15013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:14:07.466538 15013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 2 LEADER]: Becoming Leader. State: Replica: 62ec900a2cbc4383b9f34e48e4fd54c6, State: Running, Role: LEADER
I20260810 18:14:07.466840 15013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:07.467505 15007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:14:07.467949 15013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ec900a2cbc4383b9f34e48e4fd54c6. Latest consensus state: current_term: 2 leader_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } } }
I20260810 18:14:07.468062 15013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:07.468511 15017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:14:07.468672 15014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } } }
I20260810 18:14:07.468850 15014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:07.473341 15017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:14:07.474006 15017 catalog_manager.cc:1259] Loaded cluster ID: a688a7d20a384a3faf3cd7514d5c775d
I20260810 18:14:07.474058 15017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:14:07.475143 15017 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 18:14:07.477954 15017 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 18:14:07.482446 15017 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:14:07.483208 15017 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Loaded TSK: 0
I20260810 18:14:07.484328 15017 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:14:07.559818 15009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:07.560164 15009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:07.560225 15009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:07.564208 15009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:07.564262 15009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:07.564318 15009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:14:07.569495 15009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:35353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=36323
--tserver_master_addrs=127.23.65.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:07.570515 15009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:07.571516 15009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:07.581883 15048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:07.581876 15051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:07.582096 15009 server_base.cc:1034] running on GCE node
W20260810 18:14:07.582371 15054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:07.582765 15009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:07.583429 15009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:07.584590 15009 hybrid_clock.cc:648] HybridClock initialized: now 1786385647584555 us; error 53 us; skew 500 ppm
Aug 10 18:14:07 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:07.587366 15009 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:14:07.588255 15009 webserver.cc:460] Webserver started at http://127.23.65.65:36323/ using document root <none> and password file <none>
I20260810 18:14:07.588796 15009 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:07.588881 15009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:07.592823 15009 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 18:14:07.594969 15063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:07.596347 15009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:07.596454 15009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "6ad4e02f99294f948529880658da3f4b"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "b9d1a8533a3a39e978f09673bb612b7f"
server_key_iv: "d3f1f3ad07a819665d85a5c768cc8834"
server_key_version: "encryptionkey@0"
I20260810 18:14:07.596976 15009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:07.610026 15009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:07.612996 15009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:07.613176 15009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:07.613749 15009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:07.614873 15078 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:14:07.619081 15009 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:14:07.619143 15009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260810 18:14:07.619191 15009 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:14:07.620913 15009 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:14:07.620976 15009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 18:14:07.621254 15078 tablet_bootstrap.cc:492] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Bootstrap starting.
I20260810 18:14:07.631817 15009 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:35353
I20260810 18:14:07.631830 15248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:35353 every 8 connection(s)
I20260810 18:14:07.633034 15009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:14:07.640071 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 15009
I20260810 18:14:07.640601 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:40565
--local_ip_for_outbound_sockets=127.23.65.66
--tserver_master_addrs=127.23.65.126:46443
--webserver_port=34665
--webserver_interface=127.23.65.66
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:14:07.642796 15078 log.cc:824] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:14:07 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:07.650758 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.635182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:45613 (local address 127.23.65.126:46443)
0810 18:14:07.635594 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:07.635611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:07.636027 (+ 416us) server_negotiation.cc:407] Connection header received
0810 18:14:07.637139 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:07.637151 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:07.637451 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:07.637765 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:07.638904 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.639946 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.641278 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.641761 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.641925 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.644778 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:07.644785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:07.644801 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:07.644844 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:07.647005 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:07.647672 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:07.647678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:07.647683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:07.647838 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:07.648281 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:07.648285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:07.648286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:07.648764 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:07.648901 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:07.649606 (+ 705us) server_negotiation.cc:299] Negotiation successful
0810 18:14:07.649916 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":112,"threads_started":1}
I20260810 18:14:07.651067 15078 tablet_bootstrap.cc:492] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:14:07.651142 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.635314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:45613)
0810 18:14:07.635862 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:07.635905 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:07.636733 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:07.637827 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:07.637835 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:07.638270 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:07.638736 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.638750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.640095 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.640098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.641156 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.641167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.641904 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.641908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.642096 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.642713 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:07.642735 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:07.644568 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:07.647134 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:07.647140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:07.647152 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:07.647500 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:07.647950 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:07.647954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:07.647956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:07.648159 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:07.648889 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:07.648896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:07.649215 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 18:14:07.649426 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 18:14:07.649814 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":200,"threads_started":1}
I20260810 18:14:07.651484 15078 tablet_bootstrap.cc:492] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Bootstrap complete.
I20260810 18:14:07.652822 15078 ts_tablet_manager.cc:1397] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Time spent bootstrapping tablet: real 0.032s user 0.030s sys 0.000s
I20260810 18:14:07.653869 15249 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46443
I20260810 18:14:07.654103 15249 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:07.654640 15249 heartbeater.cc:511] Master 127.23.65.126:46443 requested a full tablet report, sending...
I20260810 18:14:07.655449 15078 raft_consensus.cc:348] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:07.655980 15078 raft_consensus.cc:729] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad4e02f99294f948529880658da3f4b, State: Initialized, Role: FOLLOWER
I20260810 18:14:07.656448 15078 consensus_queue.cc:260] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } }
I20260810 18:14:07.657399 14949 ts_manager.cc:194] Registered new tserver with Master: 6ad4e02f99294f948529880658da3f4b (127.23.65.65:35353)
I20260810 18:14:07.657466 15078 ts_tablet_manager.cc:1428] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:14:07.660717 14949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:45613
I20260810 18:14:07.663053 15249 heartbeater.cc:503] Master 127.23.65.126:46443 was elected leader, sending a full tablet report...
Aug 10 18:14:07 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:14:07.670449 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.660108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:35353 (local address 127.0.0.1:58904)
0810 18:14:07.660902 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:07.660949 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:07.661083 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:07.661565 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:07.661571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:07.662003 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:07.662132 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.662140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.663380 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.663384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.664379 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.664390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.664952 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.664956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.665063 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.665578 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:07.665593 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:07.667083 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:07.668854 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:07.668862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:07.668866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:07.669109 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:07.669413 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:07.669414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:07.669416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:07.669486 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:07.669979 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:07.669983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:07.670076 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 18:14:07.670180 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 18:14:07.670324 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":109,"threads_started":1}
I20260810 18:14:07.670598 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.660209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.23.65.65:35353)
0810 18:14:07.660564 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:07.660575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:07.661044 (+ 469us) server_negotiation.cc:407] Connection header received
0810 18:14:07.661234 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:07.661239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:07.661432 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:07.661566 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:07.662256 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.663265 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.664483 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.664842 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.664926 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.667211 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:07.667220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:07.667229 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:07.667250 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:07.668729 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:07.669219 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:07.669225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:07.669227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:07.669330 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:07.669557 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:07.669560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:07.669562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:07.669881 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:07.670011 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:07.670276 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 18:14:07.670442 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
I20260810 18:14:07.671495 15140 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f9915f45844d80ad44882d82a8c205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:07 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:58904
I20260810 18:14:07.672312 15262 tablet_replica.cc:330] stopping tablet replica
I20260810 18:14:07.672569 15262 raft_consensus.cc:2229] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:14:07.672787 15262 raft_consensus.cc:2258] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:07.673641 15262 ts_tablet_manager.cc:1905] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:07.677007 15262 ts_tablet_manager.cc:1918] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:07.677088 15262 log.cc:1196] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/64f9915f45844d80ad44882d82a8c205
I20260810 18:14:07.677299 15262 ts_tablet_manager.cc:1939] T 64f9915f45844d80ad44882d82a8c205 P 6ad4e02f99294f948529880658da3f4b: Deleting consensus metadata
I20260810 18:14:07.678745 14922 catalog_manager.cc:4841] TS 6ad4e02f99294f948529880658da3f4b (127.23.65.65:35353): tablet 64f9915f45844d80ad44882d82a8c205 (table test-table [id=443eb879e3de402d95156f04eda46d2f]) successfully deleted
W20260810 18:14:07.758369 15253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:07.758622 15253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:07.758679 15253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:07.762619 15253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:07.762682 15253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:07.762759 15253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:14:07.767895 15253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:40565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=34665
--tserver_master_addrs=127.23.65.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:07.768908 15253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:07.769939 15253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:07.779227 15268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:07.779228 15267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:07.779315 15253 server_base.cc:1034] running on GCE node
W20260810 18:14:07.779235 15273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:07.779891 15253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:07.780467 15253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:07.781647 15253 hybrid_clock.cc:648] HybridClock initialized: now 1786385647781600 us; error 39 us; skew 500 ppm
Aug 10 18:14:07 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:07.784211 15253 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:14:07.785034 15253 webserver.cc:460] Webserver started at http://127.23.65.66:34665/ using document root <none> and password file <none>
I20260810 18:14:07.785538 15253 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:07.785630 15253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:07.789232 15253 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:14:07.791143 15287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:07.792268 15253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:07.792373 15253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "df806df0b0064f1e97583f7a0d798e83"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "d1b62f54d339936e9fc87ea67e50e9e1"
server_key_iv: "e56a8db09d49c1147ccb0eee60df1030"
server_key_version: "encryptionkey@0"
I20260810 18:14:07.792881 15253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:07.803519 15253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:07.806000 15253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:07.806167 15253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:07.806691 15253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:07.807436 15253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:07.807483 15253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:07.807577 15253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:07.807606 15253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:07.815841 15253 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:40565
I20260810 18:14:07.815874 15460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:40565 every 8 connection(s)
I20260810 18:14:07.816824 15253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:14:07.820046 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 15253
I20260810 18:14:07.820436 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:45379
--local_ip_for_outbound_sockets=127.23.65.67
--tserver_master_addrs=127.23.65.126:46443
--webserver_port=33025
--webserver_interface=127.23.65.67
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:14:07 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:07.829939 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.818504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:40169 (local address 127.23.65.126:46443)
0810 18:14:07.818629 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:07.818633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:07.819441 (+ 808us) server_negotiation.cc:407] Connection header received
0810 18:14:07.820813 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:07.820817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:07.820883 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:07.820955 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:07.822087 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.822611 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.823735 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.824057 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.824158 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.826418 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:07.826422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:07.826425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:07.826456 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:07.827975 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:07.828442 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:07.828445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:07.828447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:07.828542 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:07.828833 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:07.828836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:07.828838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:07.829068 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:07.829172 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:07.829618 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 18:14:07.829757 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:14:07.830587 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:07.818865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:40169)
0810 18:14:07.819299 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:07.819340 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:07.820604 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:07.821133 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:07.821140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:07.821483 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:07.821941 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.821955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.822732 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.822735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.823612 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:07.823623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:07.824214 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:07.824217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:07.824364 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:07.824881 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:07.824897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:07.826250 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:07.828085 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:07.828090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:07.828100 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:07.828349 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:07.828625 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:07.828628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:07.828630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:07.828751 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:07.829168 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:07.829172 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:07.829388 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 18:14:07.829532 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 18:14:07.829782 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260810 18:14:07.831732 15462 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46443
I20260810 18:14:07.831996 15462 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:07.832566 15462 heartbeater.cc:511] Master 127.23.65.126:46443 requested a full tablet report, sending...
I20260810 18:14:07.833653 14949 ts_manager.cc:194] Registered new tserver with Master: df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565)
I20260810 18:14:07.834185 14949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:40169
W20260810 18:14:07.931978 15465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:07.932220 15465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:07.932286 15465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:07.936224 15465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:07.936286 15465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:07.936359 15465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:14:07.941383 15465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:45379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=33025
--tserver_master_addrs=127.23.65.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:07.942327 15465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:07.943305 15465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:07.952214 15474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:07.952198 15470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:07.952235 15472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:07.952494 15465 server_base.cc:1034] running on GCE node
I20260810 18:14:07.953011 15465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:07.953585 15465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:07.954761 15465 hybrid_clock.cc:648] HybridClock initialized: now 1786385647954739 us; error 31 us; skew 500 ppm
Aug 10 18:14:07 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:07.957361 15465 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:14:07.958144 15465 webserver.cc:460] Webserver started at http://127.23.65.67:33025/ using document root <none> and password file <none>
I20260810 18:14:07.958644 15465 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:07.958719 15465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:07.962409 15465 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:07.964351 15487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:07.965446 15465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:07.965551 15465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "5049583e02a1442d8cb4f39caa5eb0f7"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "29da7f3c719a535aea08332280834cb9"
server_key_iv: "3b28ff2c30012a4235ed55458bbc1877"
server_key_version: "encryptionkey@0"
I20260810 18:14:07.966037 15465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:07.993993 15465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:07.996898 15465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:07.997082 15465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:07.997644 15465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:07.998450 15465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:07.998500 15465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:07.998554 15465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:07.998579 15465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:08.007296 15465 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:45379
I20260810 18:14:08.007332 15683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:45379 every 8 connection(s)
I20260810 18:14:08.008288 15465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
Aug 10 18:14:08 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:08.018126 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 15465
I20260810 18:14:08.021590 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.010069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:46311 (local address 127.23.65.126:46443)
0810 18:14:08.010203 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:08.010206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:08.011014 (+ 808us) server_negotiation.cc:407] Connection header received
0810 18:14:08.012371 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:08.012375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:08.012433 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:08.012510 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:08.013644 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.014164 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.015298 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.015522 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.015646 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.018029 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:08.018033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:08.018036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:08.018063 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:08.019506 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:08.020107 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:08.020111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:08.020113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:08.020211 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:08.020530 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:08.020533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:08.020535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:08.020751 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:08.020840 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:08.021369 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 18:14:08.021469 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:14:08.022446 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.010426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:46311)
0810 18:14:08.010867 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.010908 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:08.012161 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:08.012687 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:08.012693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:08.013041 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:08.013503 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.013516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.014283 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.014286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.015172 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.015182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.015692 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.015696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.015866 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.016375 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:08.016391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:08.017863 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:08.019635 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:08.019640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:08.019651 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:08.020016 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:08.020313 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:08.020315 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:08.020317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:08.020449 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:08.020848 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:08.020854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:08.021143 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 18:14:08.021341 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 18:14:08.021655 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260810 18:14:08.023654 15684 heartbeater.cc:347] Connected to a master server at 127.23.65.126:46443
I20260810 18:14:08.023898 15684 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:08.024436 15684 heartbeater.cc:511] Master 127.23.65.126:46443 requested a full tablet report, sending...
I20260810 18:14:08.025663 14949 ts_manager.cc:194] Registered new tserver with Master: 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
I20260810 18:14:08.026343 14949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:46311
I20260810 18:14:08.027395 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.018790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.23.65.126:46443)
0810 18:14:08.019038 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:08.019042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:08.019792 (+ 750us) server_negotiation.cc:407] Connection header received
0810 18:14:08.019885 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:08.019888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:08.019940 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:08.020041 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:08.020792 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.021546 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.022547 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.022870 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.022960 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.024325 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:08.024329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:08.024332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:08.024356 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:08.025830 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:08.026249 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:08.026252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:08.026253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:08.026319 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:08.026568 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:08.026571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:08.026573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:08.026783 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:08.026973 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:08.027117 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 18:14:08.027237 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
I20260810 18:14:08.029023 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:14:08.033066 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.029598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.23.65.126:46443)
0810 18:14:08.029794 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:08.029797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:08.029874 (+ 77us) server_negotiation.cc:407] Connection header received
0810 18:14:08.030028 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:08.030031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:08.030115 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:08.030197 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:14:08.030440 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.030915 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.031768 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.031979 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.032062 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.032361 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:14:08.032696 (+ 335us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:14:08.032804 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:08.032893 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 18:14:08.032949 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260810 18:14:08.033715 14949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:14:08.036911 14949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:14:08.043885 15140 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf08a6bcc3248ba99a274885eb0b798 (DEFAULT_TABLE table=test-table [id=1aa756a51ed74b2ab7a653499601462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:08.044580 15140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf08a6bcc3248ba99a274885eb0b798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:08.047559 15078 tablet_bootstrap.cc:492] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Bootstrap starting.
I20260810 18:14:08.048456 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.042859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:40565 (local address 127.0.0.1:54814)
0810 18:14:08.043013 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.043027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:08.043097 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:08.044180 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:08.044184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:08.044203 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:08.044307 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.044313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.045605 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.045608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.046784 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.046795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.048029 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.048033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.048135 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.048186 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 18:14:08.048240 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 18:14:08.048324 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":16}
I20260810 18:14:08.048468 15078 tablet_bootstrap.cc:654] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:08.049098 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.043294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.23.65.66:40565)
0810 18:14:08.043766 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:08.043776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:08.043795 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:14:08.043859 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:08.043863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:08.044024 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:08.044458 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:08.044588 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.045460 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.046921 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.047761 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.048635 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.048815 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:08.048870 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 18:14:08.048960 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":111,"threads_started":1}
I20260810 18:14:08.049098 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.043096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:45379 (local address 127.0.0.1:56962)
0810 18:14:08.043501 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.043514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:08.043632 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:08.044259 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:08.044261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:08.044276 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:08.044342 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.044347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.045616 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.045619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.047938 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.047948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.048734 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.048737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.048820 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.048842 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:14:08.048901 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 18:14:08.048966 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":2,"thread_start_us":116,"threads_started":1}
I20260810 18:14:08.049154 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.043355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.23.65.67:45379)
0810 18:14:08.043780 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:08.043792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:08.043814 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:14:08.043884 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:08.043888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:08.044075 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:08.044230 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:08.044442 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.045466 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.048065 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.048639 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.048734 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.048889 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:08.048936 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 18:14:08.049025 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":95,"threads_started":1}
I20260810 18:14:08.050465 15078 tablet_bootstrap.cc:492] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: No bootstrap required, opened a new log
I20260810 18:14:08.050561 15078 ts_tablet_manager.cc:1397] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:14:08.050999 15078 raft_consensus.cc:348] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.051120 15078 raft_consensus.cc:374] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:08.051164 15078 raft_consensus.cc:729] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad4e02f99294f948529880658da3f4b, State: Initialized, Role: FOLLOWER
I20260810 18:14:08.051294 15078 consensus_queue.cc:260] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.051697 15078 ts_tablet_manager.cc:1428] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:14:08.051944 15568 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf08a6bcc3248ba99a274885eb0b798 (DEFAULT_TABLE table=test-table [id=1aa756a51ed74b2ab7a653499601462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:08.051927 15363 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf08a6bcc3248ba99a274885eb0b798 (DEFAULT_TABLE table=test-table [id=1aa756a51ed74b2ab7a653499601462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:08.052901 15568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf08a6bcc3248ba99a274885eb0b798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:08.052966 15363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf08a6bcc3248ba99a274885eb0b798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:08.059283 15705 tablet_bootstrap.cc:492] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: Bootstrap starting.
I20260810 18:14:08.059286 15706 tablet_bootstrap.cc:492] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: Bootstrap starting.
I20260810 18:14:08.061540 15705 tablet_bootstrap.cc:654] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:08.061539 15706 tablet_bootstrap.cc:654] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:08.062322 15705 log.cc:824] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:08.062326 15706 log.cc:824] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:08.064311 15705 tablet_bootstrap.cc:492] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: No bootstrap required, opened a new log
I20260810 18:14:08.064512 15705 ts_tablet_manager.cc:1397] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:14:08.064855 15706 tablet_bootstrap.cc:492] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: No bootstrap required, opened a new log
I20260810 18:14:08.065037 15706 ts_tablet_manager.cc:1397] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:14:08.068431 15705 raft_consensus.cc:348] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.068707 15705 raft_consensus.cc:374] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:08.068768 15705 raft_consensus.cc:729] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df806df0b0064f1e97583f7a0d798e83, State: Initialized, Role: FOLLOWER
I20260810 18:14:08.068727 15706 raft_consensus.cc:348] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.069005 15706 raft_consensus.cc:374] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:08.069072 15706 raft_consensus.cc:729] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5049583e02a1442d8cb4f39caa5eb0f7, State: Initialized, Role: FOLLOWER
I20260810 18:14:08.069237 15705 consensus_queue.cc:260] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.069465 15706 consensus_queue.cc:260] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.069993 15462 heartbeater.cc:503] Master 127.23.65.126:46443 was elected leader, sending a full tablet report...
I20260810 18:14:08.070140 15684 heartbeater.cc:503] Master 127.23.65.126:46443 was elected leader, sending a full tablet report...
I20260810 18:14:08.070327 15705 ts_tablet_manager.cc:1428] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:14:08.070329 15706 ts_tablet_manager.cc:1428] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
W20260810 18:14:08.134336 15250 tablet.cc:2369] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:14:08.188234 15255 raft_consensus.cc:482] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:14:08.188498 15255 raft_consensus.cc:504] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.189644 15255 leader_election.cc:288] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565), 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
I20260810 18:14:08.194591 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.190036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:47673 (local address 127.23.65.67:45379)
0810 18:14:08.190168 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:08.190173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:08.190317 (+ 144us) server_negotiation.cc:407] Connection header received
0810 18:14:08.190502 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:08.190505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:08.190555 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:08.190647 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:08.190971 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.191793 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.193532 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.194013 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.194126 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.194181 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:08.194387 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 18:14:08.194467 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:14:08.194613 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.189892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:45379 (local address 127.23.65.65:47673)
0810 18:14:08.190228 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.190244 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:08.190380 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:08.190668 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:08.190672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:08.190717 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:08.190867 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.190875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.191915 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.191919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.193420 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.193431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.194129 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.194132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.194228 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.194283 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 18:14:08.194361 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:14:08.194443 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":155,"spinlock_wait_cycles":4352,"thread_start_us":94,"threads_started":1}
I20260810 18:14:08.194674 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.190024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:35099 (local address 127.23.65.66:40565)
0810 18:14:08.190168 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:08.190173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:08.190361 (+ 188us) server_negotiation.cc:407] Connection header received
0810 18:14:08.190502 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:08.190505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:08.190558 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:08.190645 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:08.190992 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.191805 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.193610 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.194129 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.194241 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.194291 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:08.194489 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 18:14:08.194567 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:14:08.195092 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.189892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:40565 (local address 127.23.65.65:35099)
0810 18:14:08.190292 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.190304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:08.190380 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:08.190668 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:08.190672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:08.190715 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:08.190871 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.190876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.191915 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.191919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.193418 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:08.193431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.194246 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.194250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:08.194359 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.194378 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:14:08.194850 (+ 472us) client_negotiation.cc:240] Negotiation successful
0810 18:14:08.194937 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":186,"spinlock_wait_cycles":32512,"thread_start_us":72,"threads_started":1}
I20260810 18:14:08.195310 15602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf08a6bcc3248ba99a274885eb0b798" candidate_uuid: "6ad4e02f99294f948529880658da3f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" is_pre_election: true
I20260810 18:14:08.195674 15602 raft_consensus.cc:2454] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad4e02f99294f948529880658da3f4b in term 0.
I20260810 18:14:08.195735 15393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf08a6bcc3248ba99a274885eb0b798" candidate_uuid: "6ad4e02f99294f948529880658da3f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df806df0b0064f1e97583f7a0d798e83" is_pre_election: true
I20260810 18:14:08.196043 15393 raft_consensus.cc:2454] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad4e02f99294f948529880658da3f4b in term 0.
I20260810 18:14:08.196156 15069 leader_election.cc:302] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5049583e02a1442d8cb4f39caa5eb0f7, 6ad4e02f99294f948529880658da3f4b; no voters:
I20260810 18:14:08.196447 15255 raft_consensus.cc:2781] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:14:08.196548 15255 raft_consensus.cc:482] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:14:08.196599 15255 raft_consensus.cc:3037] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:08.197660 15255 raft_consensus.cc:504] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.197997 15255 leader_election.cc:288] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 1 election: Requested vote from peers df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565), 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
I20260810 18:14:08.198367 15602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf08a6bcc3248ba99a274885eb0b798" candidate_uuid: "6ad4e02f99294f948529880658da3f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5049583e02a1442d8cb4f39caa5eb0f7"
I20260810 18:14:08.198355 15393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf08a6bcc3248ba99a274885eb0b798" candidate_uuid: "6ad4e02f99294f948529880658da3f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df806df0b0064f1e97583f7a0d798e83"
I20260810 18:14:08.198493 15393 raft_consensus.cc:3037] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:08.198493 15602 raft_consensus.cc:3037] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:08.199822 15393 raft_consensus.cc:2454] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad4e02f99294f948529880658da3f4b in term 1.
I20260810 18:14:08.199822 15602 raft_consensus.cc:2454] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad4e02f99294f948529880658da3f4b in term 1.
I20260810 18:14:08.200111 15068 leader_election.cc:302] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5049583e02a1442d8cb4f39caa5eb0f7, 6ad4e02f99294f948529880658da3f4b, df806df0b0064f1e97583f7a0d798e83; no voters:
I20260810 18:14:08.200289 15255 raft_consensus.cc:2781] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:14:08.200466 15255 raft_consensus.cc:686] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 LEADER]: Becoming Leader. State: Replica: 6ad4e02f99294f948529880658da3f4b, State: Running, Role: LEADER
I20260810 18:14:08.200734 15255 consensus_queue.cc:237] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.203509 14949 catalog_manager.cc:5495] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ad4e02f99294f948529880658da3f4b (127.23.65.65). New cstate: current_term: 1 leader_uuid: "6ad4e02f99294f948529880658da3f4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } health_report { overall_health: HEALTHY } } }
I20260810 18:14:08.239708 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.236142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.23.65.65:35353)
0810 18:14:08.236436 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:08.236442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:08.236454 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:14:08.236526 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:08.236528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:08.236580 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:08.236687 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:14:08.236934 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.237447 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.238249 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:08.238470 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:08.238558 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:08.238835 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:14:08.239271 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:14:08.239373 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:08.239465 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 18:14:08.239523 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":108,"threads_started":1}
I20260810 18:14:08.243865 15393 raft_consensus.cc:1261] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [term 1 FOLLOWER]: Refusing update from remote peer 6ad4e02f99294f948529880658da3f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:08.243865 15602 raft_consensus.cc:1261] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 1 FOLLOWER]: Refusing update from remote peer 6ad4e02f99294f948529880658da3f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:08.244669 15255 consensus_queue.cc:1035] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:08.244895 15716 consensus_queue.cc:1035] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:08.252756 15726 mvcc.cc:205] Tried to move back new op lower bound from 7317035615199555584 to 7317035615032598528. Current Snapshot: MvccSnapshot[applied={T|T < 7317035615199555584}]
I20260810 18:14:08.255415 15727 mvcc.cc:205] Tried to move back new op lower bound from 7317035615199555584 to 7317035615032598528. Current Snapshot: MvccSnapshot[applied={T|T < 7317035615199555584}]
I20260810 18:14:08.259893 14947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:14:08.260041 14947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:14:08.262121 14947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Sending DeleteTablet for 3 replicas of tablet 9cf08a6bcc3248ba99a274885eb0b798
I20260810 18:14:08.262847 15568 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf08a6bcc3248ba99a274885eb0b798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:08 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:56962
I20260810 18:14:08.262852 15363 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf08a6bcc3248ba99a274885eb0b798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:08 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:54814
I20260810 18:14:08.263051 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 15009
I20260810 18:14:08.270846 15741 tablet_replica.cc:330] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: stopping tablet replica
I20260810 18:14:08.271189 15741 raft_consensus.cc:2229] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:14:08.271522 15741 raft_consensus.cc:2258] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:08.271924 15742 tablet_replica.cc:330] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: stopping tablet replica
I20260810 18:14:08.272202 15742 raft_consensus.cc:2229] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:14:08.272475 15742 raft_consensus.cc:2258] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:08.274041 15742 ts_tablet_manager.cc:1905] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:08.274041 15741 ts_tablet_manager.cc:1905] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:08.276800 15742 ts_tablet_manager.cc:1918] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:08.276800 15741 ts_tablet_manager.cc:1918] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:08.276921 15741 log.cc:1196] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/9cf08a6bcc3248ba99a274885eb0b798
I20260810 18:14:08.276994 15742 log.cc:1196] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/9cf08a6bcc3248ba99a274885eb0b798
I20260810 18:14:08.277318 15742 ts_tablet_manager.cc:1939] T 9cf08a6bcc3248ba99a274885eb0b798 P 5049583e02a1442d8cb4f39caa5eb0f7: Deleting consensus metadata
I20260810 18:14:08.277345 15741 ts_tablet_manager.cc:1939] T 9cf08a6bcc3248ba99a274885eb0b798 P df806df0b0064f1e97583f7a0d798e83: Deleting consensus metadata
I20260810 18:14:08.278692 14922 catalog_manager.cc:4841] TS df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565): tablet 9cf08a6bcc3248ba99a274885eb0b798 (table test-table [id=1aa756a51ed74b2ab7a653499601462b]) successfully deleted
I20260810 18:14:08.278782 14925 catalog_manager.cc:4841] TS 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379): tablet 9cf08a6bcc3248ba99a274885eb0b798 (table test-table [id=1aa756a51ed74b2ab7a653499601462b]) successfully deleted
W20260810 18:14:08.279627 14922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:35353 (error 108)
I20260810 18:14:08.279903 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 15253
I20260810 18:14:08.283025 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.282583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:35353 (local address 127.0.0.1:58916)
0810 18:14:08.282770 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.282875 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:35353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 18:14:08.283281 14922 catalog_manager.cc:4584] TS 6ad4e02f99294f948529880658da3f4b (127.23.65.65:35353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cf08a6bcc3248ba99a274885eb0b798: Network error: Client connection negotiation failed: client connection to 127.23.65.65:35353: connect: Connection refused (error 111)
I20260810 18:14:08.294075 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 15465
I20260810 18:14:08.305944 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 14896
I20260810 18:14:08.318985 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46443
--webserver_interface=127.23.65.126
--webserver_port=34211
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:46443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:08.430938 15745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:08.431201 15745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:08.431257 15745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:08.435202 15745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:14:08.435261 15745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:08.435288 15745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:08.435308 15745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:14:08.435329 15745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:14:08.440485 15745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:46443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:46443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:08.441499 15745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:08.442502 15745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:08.450855 15752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:08.450907 15753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:08.450955 15745 server_base.cc:1034] running on GCE node
W20260810 18:14:08.450836 15757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:08.451740 15745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:08.452768 15745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:08.453923 15745 hybrid_clock.cc:648] HybridClock initialized: now 1786385648453909 us; error 33 us; skew 500 ppm
Aug 10 18:14:08 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385648, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:08.456530 15745 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:14:08.457324 15745 webserver.cc:460] Webserver started at http://127.23.65.126:34211/ using document root <none> and password file <none>
I20260810 18:14:08.457803 15745 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:08.457859 15745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:08.461504 15745 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:14:08.463526 15769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:08.464742 15745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:08.464843 15745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "62ec900a2cbc4383b9f34e48e4fd54c6"
format_stamp: "Formatted at 2026-08-10 18:14:05 on dist-test-slave-jpj1"
server_key: "a94b13788e963b21c5f78563257d404d"
server_key_iv: "06d529849d30ee99e8268dc75aa30652"
server_key_version: "encryptionkey@0"
I20260810 18:14:08.465346 15745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:08.476693 15745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:08.479133 15745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:08.479282 15745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:08.486066 15745 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:46443
I20260810 18:14:08.486335 15845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:46443 every 8 connection(s)
I20260810 18:14:08.487092 15745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:14:08.491250 15846 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:14:08.492908 15846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Bootstrap starting.
I20260810 18:14:08.495225 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 15745
I20260810 18:14:08.495682 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:35353
--local_ip_for_outbound_sockets=127.23.65.65
--tserver_master_addrs=127.23.65.126:46443
--webserver_port=36323
--webserver_interface=127.23.65.65
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:14:08.514612 15846 log.cc:824] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:08.528981 15846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:14:08.529426 15846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: Bootstrap complete.
I20260810 18:14:08.534895 15846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:08.535406 15846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ec900a2cbc4383b9f34e48e4fd54c6, State: Initialized, Role: FOLLOWER
I20260810 18:14:08.535985 15846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:08.536134 15846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:14:08.536202 15846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:14:08.536306 15846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:14:08.538857 15846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:08.539234 15846 leader_election.cc:302] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ec900a2cbc4383b9f34e48e4fd54c6; no voters:
I20260810 18:14:08.539685 15846 leader_election.cc:288] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 18:14:08.539762 15850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:14:08.539999 15850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [term 3 LEADER]: Becoming Leader. State: Replica: 62ec900a2cbc4383b9f34e48e4fd54c6, State: Running, Role: LEADER
I20260810 18:14:08.540323 15850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } }
I20260810 18:14:08.540930 15846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:14:08.541765 15851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } } }
I20260810 18:14:08.541896 15851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:08.542128 15853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ec900a2cbc4383b9f34e48e4fd54c6. Latest consensus state: current_term: 3 leader_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ec900a2cbc4383b9f34e48e4fd54c6" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 46443 } } }
I20260810 18:14:08.542217 15853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:08.545848 15874 catalog_manager.cc:1259] Loaded cluster ID: a688a7d20a384a3faf3cd7514d5c775d
I20260810 18:14:08.545898 15874 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: loading cluster ID for follower catalog manager: success
W20260810 18:14:08.546756 15874 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 62ec900a2cbc4383b9f34e48e4fd54c6: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260810 18:14:08.546816 15874 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260810 18:14:08.546866 15864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:14:08.549000 15864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:14:08.549427 15864 catalog_manager.cc:1259] Loaded cluster ID: a688a7d20a384a3faf3cd7514d5c775d
I20260810 18:14:08.549477 15864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 18:14:08.550035 15864 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4c05ea09ad google::LogMessage::Fail() at ??:0
@ 0x7f4c05ea388a google::LogMessage::SendToLog() at ??:0
@ 0x7f4c05ea04de google::LogMessage::Flush() at ??:0
@ 0x7f4c05ea1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4c107e3fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4c107e4d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4c107e1579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4c10825640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4c080f23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4c080f2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4c080f43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4c080e3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4c097276db start_thread at ??:0
@ 0x7f4c0499e61f clone at ??:0
W20260810 18:14:08.635143 15848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:08.635385 15848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:08.635443 15848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:08.639354 15848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:08.639410 15848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:08.639487 15848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:14:08.644691 15848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:35353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=36323
--tserver_master_addrs=127.23.65.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:08.645699 15848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:08.646682 15848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:08.656277 15918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:08.656277 15916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:08.656522 15848 server_base.cc:1034] running on GCE node
W20260810 18:14:08.656337 15920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:08.657079 15848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:08.657709 15848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:08.658879 15848 hybrid_clock.cc:648] HybridClock initialized: now 1786385648658852 us; error 44 us; skew 500 ppm
Aug 10 18:14:08 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385648, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:08.661816 15848 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:14:08.662938 15848 webserver.cc:460] Webserver started at http://127.23.65.65:36323/ using document root <none> and password file <none>
I20260810 18:14:08.663679 15848 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:08.663771 15848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:08.668815 15848 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:14:08.671402 15927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:08.672984 15848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:14:08.673105 15848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "6ad4e02f99294f948529880658da3f4b"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "b9d1a8533a3a39e978f09673bb612b7f"
server_key_iv: "d3f1f3ad07a819665d85a5c768cc8834"
server_key_version: "encryptionkey@0"
I20260810 18:14:08.673799 15848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:08.695760 15848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:08.699334 15848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:08.699574 15848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:08.700333 15848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:08.701730 15940 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:14:08.705782 15848 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:14:08.705856 15848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260810 18:14:08.705919 15848 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:14:08.708055 15848 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:14:08.708113 15848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:08.708388 15940 tablet_bootstrap.cc:492] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Bootstrap starting.
I20260810 18:14:08.719192 15848 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:35353
I20260810 18:14:08.719363 16101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:35353 every 8 connection(s)
I20260810 18:14:08.720371 15848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:14:08.724081 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 15848
I20260810 18:14:08.724562 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:40565
--local_ip_for_outbound_sockets=127.23.65.66
--tserver_master_addrs=127.23.65.126:46443
--webserver_port=34665
--webserver_interface=127.23.65.66
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:14:08.728559 15940 log.cc:824] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:08.737816 15940 tablet_bootstrap.cc:492] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:14:08.738247 15940 tablet_bootstrap.cc:492] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Bootstrap complete.
I20260810 18:14:08.739746 15940 ts_tablet_manager.cc:1397] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20260810 18:14:08.742830 15940 raft_consensus.cc:348] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.743369 15940 raft_consensus.cc:729] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad4e02f99294f948529880658da3f4b, State: Initialized, Role: FOLLOWER
I20260810 18:14:08.743916 15940 consensus_queue.cc:260] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:08.744922 15940 ts_tablet_manager.cc:1428] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260810 18:14:08.770467 16105 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:14:08.722658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:52363)
0810 18:14:08.723286 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.723356 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:08.724618 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:08.769504 (+ 44886us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":167,"threads_started":1}
W20260810 18:14:08.770987 16103 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:46443 (0 consecutive failures): Network error: Failed to ping master at 127.23.65.126:46443: Client connection negotiation failed: client connection to 127.23.65.126:46443: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 18:14:08.771764 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.771453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:47061)
0810 18:14:08.771618 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.771695 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 18:14:08.772459 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.772233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:55983)
0810 18:14:08.772347 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.772400 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 18:14:08.772631 16103 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:14:08.844118 16107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:08.844352 16107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:08.844390 16107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:08.848259 16107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:08.848307 16107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:08.848361 16107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:14:08.853407 16107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:40565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=34665
--tserver_master_addrs=127.23.65.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:08.854362 16107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:08.855289 16107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:14:08.864748 16107 server_base.cc:1034] running on GCE node
W20260810 18:14:08.864696 16122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:08.864706 16125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:08.864748 16121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:08.865330 16107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:08.865896 16107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:08.867054 16107 hybrid_clock.cc:648] HybridClock initialized: now 1786385648867035 us; error 30 us; skew 500 ppm
Aug 10 18:14:08 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385648, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:08.869678 16107 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:14:08.870477 16107 webserver.cc:460] Webserver started at http://127.23.65.66:34665/ using document root <none> and password file <none>
I20260810 18:14:08.870967 16107 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:08.871025 16107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:08.874667 16107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:14:08.876650 16139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:08.877840 16107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:14:08.877948 16107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "df806df0b0064f1e97583f7a0d798e83"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "d1b62f54d339936e9fc87ea67e50e9e1"
server_key_iv: "e56a8db09d49c1147ccb0eee60df1030"
server_key_version: "encryptionkey@0"
I20260810 18:14:08.878472 16107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:08.891800 16107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:08.894381 16107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:08.894520 16107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:08.895020 16107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:08.895816 16107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:08.895864 16107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:08.895902 16107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:08.895922 16107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:08.904255 16107 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:40565
I20260810 18:14:08.904309 16326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:40565 every 8 connection(s)
I20260810 18:14:08.905176 16107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:14:08.905992 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 16107
I20260810 18:14:08.906443 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:45379
--local_ip_for_outbound_sockets=127.23.65.67
--tserver_master_addrs=127.23.65.126:46443
--webserver_port=33025
--webserver_interface=127.23.65.67
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:14:08.909036 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.907349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:45957)
0810 18:14:08.907848 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.908337 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":137,"threads_started":1}
W20260810 18:14:08.909569 16327 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:46443 (0 consecutive failures): Network error: Failed to ping master at 127.23.65.126:46443: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
I20260810 18:14:08.910146 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.909940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:42993)
0810 18:14:08.910043 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.910092 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 18:14:08.910744 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:08.910558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:46697)
0810 18:14:08.910643 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:08.910682 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 18:14:08.910904 16327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:14:09.015982 16331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:09.016239 16331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:09.016280 16331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:09.020349 16331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:09.020408 16331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:09.020478 16331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:14:09.025694 16331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:45379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=33025
--tserver_master_addrs=127.23.65.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:09.026705 16331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:09.027705 16331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:09.037107 16338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:09.037107 16337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:09.037225 16331 server_base.cc:1034] running on GCE node
W20260810 18:14:09.037143 16340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:09.037832 16331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:09.038410 16331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:09.039592 16331 hybrid_clock.cc:648] HybridClock initialized: now 1786385649039559 us; error 41 us; skew 500 ppm
Aug 10 18:14:09 dist-test-slave-jpj1 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385649, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:09.042196 16331 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:14:09.043036 16331 webserver.cc:460] Webserver started at http://127.23.65.67:33025/ using document root <none> and password file <none>
I20260810 18:14:09.043717 16331 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:09.043814 16331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:09.048218 16331 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 18:14:09.050180 16353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:09.051358 16331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:09.051476 16331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "5049583e02a1442d8cb4f39caa5eb0f7"
format_stamp: "Formatted at 2026-08-10 18:14:06 on dist-test-slave-jpj1"
server_key: "29da7f3c719a535aea08332280834cb9"
server_key_iv: "3b28ff2c30012a4235ed55458bbc1877"
server_key_version: "encryptionkey@0"
I20260810 18:14:09.052033 16331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:09.073834 16331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:09.076864 16331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:09.077057 16331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:09.077693 16331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:09.078579 16331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:09.078631 16331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:09.078688 16331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:09.078714 16331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:09.087585 16331 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:45379
I20260810 18:14:09.087636 16523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:45379 every 8 connection(s)
I20260810 18:14:09.088577 16331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:14:09.092270 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:09.090669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:50491)
0810 18:14:09.091120 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:09.091557 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
W20260810 18:14:09.092736 16524 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:46443 (0 consecutive failures): Network error: Failed to ping master at 127.23.65.126:46443: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
I20260810 18:14:09.093320 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:09.093119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:49685)
0810 18:14:09.093216 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:09.093264 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 18:14:09.093571 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 16331
I20260810 18:14:09.093914 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:09.093725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:57973)
0810 18:14:09.093821 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:09.093864 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 18:14:09.094074 16524 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:14:09.773597 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:09.773167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:46279)
0810 18:14:09.773434 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:09.773484 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
I20260810 18:14:09.911883 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:09.911412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:36177)
0810 18:14:09.911675 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:09.911756 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260810 18:14:10.094985 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:10.094589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:47283)
0810 18:14:10.094828 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:10.094883 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260810 18:14:10.151602 16538 raft_consensus.cc:482] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:14:10.151824 16538 raft_consensus.cc:504] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:10.152935 16538 leader_election.cc:288] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565), 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
Aug 10 18:14:10 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385648, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.66@KRBTEST.COM
Aug 10 18:14:10 dist-test-slave-jpj1 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385648, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.67@KRBTEST.COM
I20260810 18:14:10.169224 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:10.153163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:40565 (local address 127.23.65.65:47533)
0810 18:14:10.153476 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:10.153491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:10.153612 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:10.155762 (+ 2150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:10.155770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:10.156475 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:10.157299 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:10.157321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:10.159214 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:10.159218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:10.160856 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:10.160866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:10.161854 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:10.161858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:10.162152 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:10.162857 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:10.162885 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:10.164675 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:10.166793 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:10.166799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:10.166813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:10.167166 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:10.167490 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:10.167493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:10.167495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:10.167688 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:10.168294 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:10.168297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:10.168531 (+ 234us) client_negotiation.cc:769] Sending connection context
0810 18:14:10.168685 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 18:14:10.169025 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":337,"spinlock_wait_cycles":61312,"thread_start_us":94,"threads_started":1}
I20260810 18:14:10.169250 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:10.153358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:47245 (local address 127.23.65.67:45379)
0810 18:14:10.153796 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:10.153823 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:10.153872 (+ 49us) server_negotiation.cc:407] Connection header received
0810 18:14:10.154803 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:10.154812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:10.155281 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:10.155662 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:10.157494 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:10.159071 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:10.160988 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:10.161701 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:10.161889 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:10.165178 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:10.165183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:10.165195 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:10.165222 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:10.166751 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:10.167271 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:10.167277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:10.167281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:10.167393 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:10.167786 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:10.167789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:10.167790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:10.168174 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:10.168297 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:10.168847 (+ 550us) server_negotiation.cc:299] Negotiation successful
0810 18:14:10.169092 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":92,"threads_started":1}
I20260810 18:14:10.169256 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:10.153163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:45379 (local address 127.23.65.65:47245)
0810 18:14:10.153476 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:10.153491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:10.153612 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:10.155762 (+ 2150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:10.155824 (+ 62us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:10.156475 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:10.157299 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:10.157321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:10.159216 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:10.159218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:10.160853 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:10.160866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:10.161854 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:10.161858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:10.162153 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:10.162881 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:10.162896 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:10.165002 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:10.166875 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:10.166878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:10.166879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:10.167156 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:10.167486 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:10.167489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:10.167491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:10.167681 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:10.168290 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:10.168296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:10.168532 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 18:14:10.168686 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 18:14:10.169101 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":540,"spinlock_wait_cycles":39936}
I20260810 18:14:10.169909 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:10.153358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:47533 (local address 127.23.65.66:40565)
0810 18:14:10.153796 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:10.153823 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:10.153872 (+ 49us) server_negotiation.cc:407] Connection header received
0810 18:14:10.154803 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:10.154812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:10.155295 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:10.155662 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:10.157494 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:10.159071 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:10.161107 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:10.161744 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:10.161889 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:10.164895 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:10.164902 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:10.164918 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:10.164953 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:10.166676 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:10.167286 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:10.167291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:10.167295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:10.167402 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:10.167787 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:10.167790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:10.167791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:10.168175 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:10.168300 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:10.168838 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 18:14:10.169084 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":90,"threads_started":1}
W20260810 18:14:10.172045 15931 leader_election.cc:341] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565): Not found: Tablet not found: 9cf08a6bcc3248ba99a274885eb0b798
W20260810 18:14:10.172206 15932 leader_election.cc:341] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379): Not found: Tablet not found: 9cf08a6bcc3248ba99a274885eb0b798
I20260810 18:14:10.172302 15932 leader_election.cc:302] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad4e02f99294f948529880658da3f4b; no voters: 5049583e02a1442d8cb4f39caa5eb0f7, df806df0b0064f1e97583f7a0d798e83
I20260810 18:14:10.172544 16538 raft_consensus.cc:2726] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:14:10.774691 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:10.774255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:37189)
0810 18:14:10.774532 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:10.774590 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260810 18:14:10.912892 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:10.912503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:48139)
0810 18:14:10.912751 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:10.912804 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260810 18:14:11.096196 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:11.095753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:47357)
0810 18:14:11.096028 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:11.096100 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260810 18:14:11.775899 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:11.775390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:51325)
0810 18:14:11.775740 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:11.775796 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":80,"threads_started":1}
I20260810 18:14:11.913937 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:11.913535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:35085)
0810 18:14:11.913788 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:11.913847 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260810 18:14:12.097419 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:12.096970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:47167)
0810 18:14:12.097249 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:12.097314 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260810 18:14:12.104734 16554 raft_consensus.cc:482] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:14:12.104874 16554 raft_consensus.cc:504] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:12.105319 16554 leader_election.cc:288] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565), 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
W20260810 18:14:12.106125 15932 leader_election.cc:341] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379): Not found: Tablet not found: 9cf08a6bcc3248ba99a274885eb0b798
W20260810 18:14:12.106230 15931 leader_election.cc:341] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565): Not found: Tablet not found: 9cf08a6bcc3248ba99a274885eb0b798
I20260810 18:14:12.106324 15931 leader_election.cc:302] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad4e02f99294f948529880658da3f4b; no voters: 5049583e02a1442d8cb4f39caa5eb0f7, df806df0b0064f1e97583f7a0d798e83
I20260810 18:14:12.106462 16554 raft_consensus.cc:2726] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:14:12.776990 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:12.776556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:48321)
0810 18:14:12.776829 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:12.776887 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260810 18:14:12.915096 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:12.914638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:36075)
0810 18:14:12.914936 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:12.915000 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":74,"threads_started":1}
I20260810 18:14:13.098559 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:13.098144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:34643)
0810 18:14:13.098408 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:13.098468 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260810 18:14:13.778388 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:13.777828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:54953)
0810 18:14:13.778178 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:13.778248 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":73,"threads_started":1}
I20260810 18:14:13.916349 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:13.915910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:47801)
0810 18:14:13.916185 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:13.916250 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260810 18:14:14.099853 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:14.099382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:51703)
0810 18:14:14.099679 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:14.099743 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":77,"threads_started":1}
I20260810 18:14:14.246120 16564 raft_consensus.cc:482] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:14:14.246265 16564 raft_consensus.cc:504] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:14.246631 16564 leader_election.cc:288] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565), 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
W20260810 18:14:14.247417 15931 leader_election.cc:341] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565): Not found: Tablet not found: 9cf08a6bcc3248ba99a274885eb0b798
W20260810 18:14:14.247552 15932 leader_election.cc:341] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379): Not found: Tablet not found: 9cf08a6bcc3248ba99a274885eb0b798
I20260810 18:14:14.247639 15932 leader_election.cc:302] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad4e02f99294f948529880658da3f4b; no voters: 5049583e02a1442d8cb4f39caa5eb0f7, df806df0b0064f1e97583f7a0d798e83
I20260810 18:14:14.247829 16564 raft_consensus.cc:2726] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:14:14.779809 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:14.779320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:56459)
0810 18:14:14.779618 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:14.779704 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":86,"threads_started":1}
I20260810 18:14:14.917465 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:14.917056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:53479)
0810 18:14:14.917303 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:14.917363 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260810 18:14:15.101001 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:15.100566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:43909)
0810 18:14:15.100838 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:15.100900 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260810 18:14:15.780937 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:15.780504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:53041)
0810 18:14:15.780778 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:15.780837 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
I20260810 18:14:15.918557 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:15.918122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:48997)
0810 18:14:15.918411 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:15.918464 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":75,"threads_started":1}
I20260810 18:14:16.102185 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:16.101704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:39341)
0810 18:14:16.102034 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:16.102092 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":95,"threads_started":1}
I20260810 18:14:16.782200 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:16.781720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:56503)
0810 18:14:16.782035 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:16.782095 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":112,"threads_started":1}
I20260810 18:14:16.919706 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:16.919258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:40689)
0810 18:14:16.919511 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:16.919605 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260810 18:14:17.103425 16579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:17.102941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:60361)
0810 18:14:17.103253 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:17.103320 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
I20260810 18:14:17.783258 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:17.782842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:35973)
0810 18:14:17.783099 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:17.783156 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260810 18:14:17.820925 16582 raft_consensus.cc:482] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:14:17.821058 16582 raft_consensus.cc:504] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df806df0b0064f1e97583f7a0d798e83" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 40565 } } peers { permanent_uuid: "5049583e02a1442d8cb4f39caa5eb0f7" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45379 } } peers { permanent_uuid: "6ad4e02f99294f948529880658da3f4b" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 35353 } }
I20260810 18:14:17.821491 16582 leader_election.cc:288] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565), 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379)
W20260810 18:14:17.822194 15931 leader_election.cc:341] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df806df0b0064f1e97583f7a0d798e83 (127.23.65.66:40565): Not found: Tablet not found: 9cf08a6bcc3248ba99a274885eb0b798
W20260810 18:14:17.822314 15932 leader_election.cc:341] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5049583e02a1442d8cb4f39caa5eb0f7 (127.23.65.67:45379): Not found: Tablet not found: 9cf08a6bcc3248ba99a274885eb0b798
I20260810 18:14:17.822383 15932 leader_election.cc:302] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad4e02f99294f948529880658da3f4b; no voters: 5049583e02a1442d8cb4f39caa5eb0f7, df806df0b0064f1e97583f7a0d798e83
I20260810 18:14:17.822512 16582 raft_consensus.cc:2726] T 9cf08a6bcc3248ba99a274885eb0b798 P 6ad4e02f99294f948529880658da3f4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:14:17.920737 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:17.920351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:58625)
0810 18:14:17.920594 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:17.920646 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260810 18:14:18.104656 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:18.104210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:32883)
0810 18:14:18.104495 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:18.104553 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":109,"threads_started":1}
I20260810 18:14:18.784538 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:18.784116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:36845)
0810 18:14:18.784376 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:18.784436 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260810 18:14:18.921868 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:18.921447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:51375)
0810 18:14:18.921702 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:18.921760 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260810 18:14:19.105820 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:19.105424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:34429)
0810 18:14:19.105667 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:19.105720 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260810 18:14:19.785632 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:19.785215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:47961)
0810 18:14:19.785472 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:19.785531 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260810 18:14:19.923030 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:19.922613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:38993)
0810 18:14:19.922871 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:19.922929 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260810 18:14:20.106997 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:20.106582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:57753)
0810 18:14:20.106839 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:20.106897 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260810 18:14:20.786768 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:20.786337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:58893)
0810 18:14:20.786602 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:20.786663 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260810 18:14:20.924230 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:20.923775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:35177)
0810 18:14:20.924067 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:20.924128 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
I20260810 18:14:21.108228 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:21.107781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:53321)
0810 18:14:21.108067 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:21.108128 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260810 18:14:21.787932 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:21.787446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:48645)
0810 18:14:21.787761 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:21.787829 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":130,"threads_started":1}
I20260810 18:14:21.925300 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:21.924880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:49855)
0810 18:14:21.925143 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:21.925200 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260810 18:14:22.109386 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:22.108933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:33361)
0810 18:14:22.109220 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:22.109282 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260810 18:14:22.789022 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:22.788603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:43809)
0810 18:14:22.788861 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:22.788921 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260810 18:14:22.926432 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:22.925964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:41637)
0810 18:14:22.926272 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:22.926333 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
I20260810 18:14:23.110524 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:23.110084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:38515)
0810 18:14:23.110362 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:23.110420 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260810 18:14:23.790352 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:23.789972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.65:55805)
0810 18:14:23.790198 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:23.790253 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260810 18:14:23.927690 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:23.927263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.66:39797)
0810 18:14:23.927515 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:23.927590 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260810 18:14:24.094626 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 15848
I20260810 18:14:24.111709 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 16107
I20260810 18:14:24.111932 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:24.111479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:46443 (local address 127.23.65.67:39313)
0810 18:14:24.111760 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:24.111822 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260810 18:14:24.123687 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 16331
2026-08-10T18:14:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19603 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:14:24 dist-test-slave-jpj1 krb5kdc[16619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:14:24 dist-test-slave-jpj1 krb5kdc[16619](info): set up 2 sockets
Aug 10 18:14:24 dist-test-slave-jpj1 krb5kdc[16619](info): commencing operation
krb5kdc: starting...
W20260810 18:14:25.291321 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:14:25 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:14:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:14:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:14:25.414418 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38855
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:25.524902 16638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:25.525166 16638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:25.525203 16638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:25.529105 16638 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 18:14:25.529153 16638 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 18:14:25.529166 16638 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 18:14:25.529179 16638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:14:25.529196 16638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:25.529223 16638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:25.529253 16638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:14:25.529276 16638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:14:25.534433 16638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:25.535431 16638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:25.536446 16638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:25.544963 16643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:25.545023 16638 server_base.cc:1034] running on GCE node
W20260810 18:14:25.544957 16647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:25.544957 16649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:25.545729 16638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:25.546875 16638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:25.548036 16638 hybrid_clock.cc:648] HybridClock initialized: now 1786385665548023 us; error 27 us; skew 500 ppm
Aug 10 18:14:25 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385665, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:25.550740 16638 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:14:25.551679 16638 webserver.cc:460] Webserver started at http://127.23.65.126:36641/ using document root <none> and password file <none>
I20260810 18:14:25.552234 16638 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:25.552305 16638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:25.552502 16638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:25.554575 16638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "6582f75204bf4effaf30f27f8948c008"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "844a81dec5f577e1751cb0585e5193f0"
server_key_iv: "f9e517d700591b68d0b84a2d3c93889e"
server_key_version: "encryptionkey@0"
I20260810 18:14:25.555130 16638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "6582f75204bf4effaf30f27f8948c008"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "844a81dec5f577e1751cb0585e5193f0"
server_key_iv: "f9e517d700591b68d0b84a2d3c93889e"
server_key_version: "encryptionkey@0"
I20260810 18:14:25.558962 16638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 18:14:25.561424 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:25.562566 16638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:25.562685 16638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "6582f75204bf4effaf30f27f8948c008"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "844a81dec5f577e1751cb0585e5193f0"
server_key_iv: "f9e517d700591b68d0b84a2d3c93889e"
server_key_version: "encryptionkey@0"
I20260810 18:14:25.562780 16638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:25.586125 16638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:25.588896 16638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:25.589075 16638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:25.596056 16638 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:38855
I20260810 18:14:25.596318 16746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:38855 every 8 connection(s)
I20260810 18:14:25.597023 16638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:14:25.599722 16747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:25.600732 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 16638
I20260810 18:14:25.600839 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:14:25.601131 23813 external_mini_cluster.cc:1428] Setting key ae60abf4efdf5dcb5f369a72747bb9da
I20260810 18:14:25.606156 16747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Bootstrap starting.
I20260810 18:14:25.608686 16747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:25.609601 16747 log.cc:824] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:14:25 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:25.611812 16747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: No bootstrap required, opened a new log
I20260810 18:14:25.615314 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:25.602544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.23.65.126:38855)
0810 18:14:25.602963 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:25.602983 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:25.603026 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:14:25.603874 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:25.603896 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:25.604302 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:25.604667 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:25.605321 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:25.606737 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:25.607607 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:25.608277 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:25.608423 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:25.611056 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:25.611063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:25.611079 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:25.611110 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:25.612713 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:25.613184 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:25.613188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:25.613192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:25.613297 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:25.613562 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:25.613564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:25.613565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:25.613930 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:25.614080 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:25.614348 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 18:14:25.614595 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260810 18:14:25.615720 16747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:14:25.615942 16747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:25.615998 16747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6582f75204bf4effaf30f27f8948c008, State: Initialized, Role: FOLLOWER
I20260810 18:14:25.616452 16747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:14:25.616587 16747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:14:25.616653 16747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:14:25.616752 16747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:25.617610 16747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:14:25.617928 16747 leader_election.cc:302] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6582f75204bf4effaf30f27f8948c008; no voters:
I20260810 18:14:25.618217 16747 leader_election.cc:288] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:14:25.618302 16754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:14:25.618520 16754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 1 LEADER]: Becoming Leader. State: Replica: 6582f75204bf4effaf30f27f8948c008, State: Running, Role: LEADER
I20260810 18:14:25.618856 16754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:14:25.619364 16747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:14:25.620544 16756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6582f75204bf4effaf30f27f8948c008. Latest consensus state: current_term: 1 leader_uuid: "6582f75204bf4effaf30f27f8948c008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } } }
I20260810 18:14:25.620674 16756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:25.621021 16755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6582f75204bf4effaf30f27f8948c008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } } }
I20260810 18:14:25.621179 16755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:25.621407 16767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:14:25.624285 16767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:14:25.629827 16767 catalog_manager.cc:1347] Generated new cluster ID: bfa272cfd09047c3bf7244ce6bf547a5
I20260810 18:14:25.629881 16767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:14:25.661549 16767 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:14:25.662549 16767 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:14:25.675760 16767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Generated new TSK 0
I20260810 18:14:25.676453 16767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:14:25.708137 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:38855
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:25.818631 16791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:25.818882 16791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:25.818944 16791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:25.822944 16791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:25.823000 16791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:25.823072 16791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:14:25.828418 16791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:25.829387 16791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:25.830360 16791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:25.839309 16801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:25.839335 16803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:25.839311 16799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:25.839596 16791 server_base.cc:1034] running on GCE node
I20260810 18:14:25.840095 16791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:25.840689 16791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:25.841861 16791 hybrid_clock.cc:648] HybridClock initialized: now 1786385665841835 us; error 43 us; skew 500 ppm
Aug 10 18:14:25 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385665, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:25.844458 16791 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:14:25.845290 16791 webserver.cc:460] Webserver started at http://127.23.65.65:42039/ using document root <none> and password file <none>
I20260810 18:14:25.845804 16791 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:25.845880 16791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:25.846081 16791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:25.848121 16791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "dc7a1a98b3684862af61b46ffcfb27be"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "05ddf81bd24f310428fc3a522469e8ae"
server_key_iv: "b79c07ce88e837db9f774996a168912c"
server_key_version: "encryptionkey@0"
I20260810 18:14:25.848659 16791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "dc7a1a98b3684862af61b46ffcfb27be"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "05ddf81bd24f310428fc3a522469e8ae"
server_key_iv: "b79c07ce88e837db9f774996a168912c"
server_key_version: "encryptionkey@0"
I20260810 18:14:25.852176 16791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:14:25.854432 16816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:25.855577 16791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:14:25.855700 16791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "dc7a1a98b3684862af61b46ffcfb27be"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "05ddf81bd24f310428fc3a522469e8ae"
server_key_iv: "b79c07ce88e837db9f774996a168912c"
server_key_version: "encryptionkey@0"
I20260810 18:14:25.855795 16791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:25.878769 16791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:25.881603 16791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:25.881786 16791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:25.882390 16791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:25.883350 16791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:25.883417 16791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:25.883519 16791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:25.883576 16791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:25.892447 16791 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:44543
I20260810 18:14:25.892475 17004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:44543 every 8 connection(s)
I20260810 18:14:25.893316 16791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:14:25.896333 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 16791
I20260810 18:14:25.896443 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:14:25.896721 23813 external_mini_cluster.cc:1428] Setting key 2ff7d231f8651b2e02d610780e43c284
Aug 10 18:14:25 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385665, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:25.907194 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:25.895028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:59013 (local address 127.23.65.126:38855)
0810 18:14:25.895173 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:25.895177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:25.896214 (+ 1037us) server_negotiation.cc:407] Connection header received
0810 18:14:25.897176 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:25.897178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:25.897233 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:25.897315 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:25.898689 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:25.899396 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:25.900863 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:25.901091 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:25.901172 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:25.903621 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:25.903626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:25.903628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:25.903657 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:25.905216 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:25.905696 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:25.905700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:25.905701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:25.905800 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:25.906101 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:25.906104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:25.906106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:25.906338 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:25.906441 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:25.906900 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 18:14:25.907029 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:14:25.907917 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:25.895366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:59013)
0810 18:14:25.896061 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:25.896103 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:25.896938 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:25.897480 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:25.897489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:25.897971 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:25.898527 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:25.898541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:25.899524 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:25.899527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:25.900720 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:25.900732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:25.901234 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:25.901238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:25.901426 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:25.902005 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:25.902022 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:25.903438 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:25.905330 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:25.905334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:25.905346 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:25.905593 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:25.905896 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:25.905898 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:25.905899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:25.906020 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:25.906430 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:25.906435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:25.906660 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 18:14:25.906806 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 18:14:25.907049 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":142,"threads_started":1}
I20260810 18:14:25.909010 17005 heartbeater.cc:347] Connected to a master server at 127.23.65.126:38855
I20260810 18:14:25.909240 17005 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:25.909813 17005 heartbeater.cc:511] Master 127.23.65.126:38855 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
I20260810 18:14:25.911439 16689 ts_manager.cc:194] Registered new tserver with Master: dc7a1a98b3684862af61b46ffcfb27be (127.23.65.65:44543)
I20260810 18:14:25.912595 16689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:59013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:14:25.925849 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:38855
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:26.037298 17012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:26.037557 17012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:26.037599 17012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:26.041589 17012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:26.041656 17012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:26.041713 17012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:14:26.046968 17012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:26.048136 17012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:26.049139 17012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:26.058537 17025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:26.058528 17023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:26.058528 17022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:26.058728 17012 server_base.cc:1034] running on GCE node
I20260810 18:14:26.059235 17012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:26.059942 17012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:26.061098 17012 hybrid_clock.cc:648] HybridClock initialized: now 1786385666061069 us; error 45 us; skew 500 ppm
Aug 10 18:14:26 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385666, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:26.063735 17012 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:14:26.064550 17012 webserver.cc:460] Webserver started at http://127.23.65.66:36445/ using document root <none> and password file <none>
I20260810 18:14:26.065073 17012 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:26.065137 17012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:26.065332 17012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:26.067355 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "6c413b3c61ae44cbac0b257c54cacbb0"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "46c73ba6f351beb7482b444e218ad0a3"
server_key_iv: "34bb6cb029e2a838e176f17b759924ee"
server_key_version: "encryptionkey@0"
I20260810 18:14:26.067951 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "6c413b3c61ae44cbac0b257c54cacbb0"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "46c73ba6f351beb7482b444e218ad0a3"
server_key_iv: "34bb6cb029e2a838e176f17b759924ee"
server_key_version: "encryptionkey@0"
I20260810 18:14:26.071509 17012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:14:26.073840 17036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:26.074986 17012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:26.075093 17012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "6c413b3c61ae44cbac0b257c54cacbb0"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "46c73ba6f351beb7482b444e218ad0a3"
server_key_iv: "34bb6cb029e2a838e176f17b759924ee"
server_key_version: "encryptionkey@0"
I20260810 18:14:26.075188 17012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:26.095041 17012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:26.098001 17012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:26.098186 17012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:26.098794 17012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:26.099677 17012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:26.099735 17012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:26.099793 17012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:26.099820 17012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:26.108304 17012 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:45489
I20260810 18:14:26.108338 17230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:45489 every 8 connection(s)
I20260810 18:14:26.109194 17012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:14:26.112421 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 17012
I20260810 18:14:26.112568 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:14:26.112900 23813 external_mini_cluster.cc:1428] Setting key 6ced118cd97b949d62016e640ba0fa89
Aug 10 18:14:26 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385666, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:14:26.123503 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.111016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:46131 (local address 127.23.65.126:38855)
0810 18:14:26.111149 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:26.111153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:26.112201 (+ 1048us) server_negotiation.cc:407] Connection header received
0810 18:14:26.113345 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:26.113348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:26.113408 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:26.113497 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:26.115032 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.115574 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.116985 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.117203 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.117293 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.119860 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:26.119863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:26.119865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:26.119890 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:26.121329 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:26.121955 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:26.121958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:26.121960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:26.122050 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:26.122374 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:26.122378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:26.122380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:26.122574 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:26.122662 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:26.123195 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 18:14:26.123336 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:14:26.124397 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.111374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:46131)
0810 18:14:26.112055 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:26.112098 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:26.113119 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:26.113838 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:26.113845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:26.114203 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:26.114883 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.114904 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.115696 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.115700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.116865 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.116875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.117347 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.117351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.117513 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.118075 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:26.118092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:26.119696 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:26.121472 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:26.121478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:26.121494 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:26.121871 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:26.122134 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:26.122137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:26.122139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:26.122281 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:26.122681 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:26.122688 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:26.122940 (+ 252us) client_negotiation.cc:769] Sending connection context
0810 18:14:26.123119 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 18:14:26.123450 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":145,"threads_started":1}
I20260810 18:14:26.125571 17231 heartbeater.cc:347] Connected to a master server at 127.23.65.126:38855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
I20260810 18:14:26.125810 17231 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:26.126291 17231 heartbeater.cc:511] Master 127.23.65.126:38855 requested a full tablet report, sending...
I20260810 18:14:26.127332 16689 ts_manager.cc:194] Registered new tserver with Master: 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489)
I20260810 18:14:26.127938 16689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:46131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:14:26.140647 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:38855
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:26.278674 17238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:26.278945 17238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:26.279003 17238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:26.282934 17238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:26.282987 17238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:26.283044 17238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:14:26.288259 17238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:26.289258 17238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:26.290275 17238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:26.300087 17247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:26.300074 17246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:26.300071 17249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:26.300554 17238 server_base.cc:1034] running on GCE node
I20260810 18:14:26.300997 17238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:26.301669 17238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:26.302853 17238 hybrid_clock.cc:648] HybridClock initialized: now 1786385666302824 us; error 48 us; skew 500 ppm
Aug 10 18:14:26 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385666, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:26.305843 17238 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:14:26.307013 17238 webserver.cc:460] Webserver started at http://127.23.65.67:38983/ using document root <none> and password file <none>
I20260810 18:14:26.307776 17238 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:26.307864 17238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:26.308106 17238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:14:26.311017 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "634a73f479994917ad451dd54ce7cddf"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "2fe0dd5a34b3296393ec19162824b43d"
server_key_iv: "a0104bcb812bd3f089f6ac3e47a3aaa7"
server_key_version: "encryptionkey@0"
I20260810 18:14:26.311774 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "634a73f479994917ad451dd54ce7cddf"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "2fe0dd5a34b3296393ec19162824b43d"
server_key_iv: "a0104bcb812bd3f089f6ac3e47a3aaa7"
server_key_version: "encryptionkey@0"
I20260810 18:14:26.316787 17238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 18:14:26.320042 17259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:26.321650 17238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:14:26.321780 17238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "634a73f479994917ad451dd54ce7cddf"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "2fe0dd5a34b3296393ec19162824b43d"
server_key_iv: "a0104bcb812bd3f089f6ac3e47a3aaa7"
server_key_version: "encryptionkey@0"
I20260810 18:14:26.321908 17238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:26.359879 17238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:26.362699 17238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:26.362857 17238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:26.363417 17238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:26.364289 17238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:26.364346 17238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:26.364404 17238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:26.364425 17238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:26.373086 17238 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:36889
I20260810 18:14:26.373114 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:36889 every 8 connection(s)
I20260810 18:14:26.373966 17238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:14:26.378073 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 17238
I20260810 18:14:26.378194 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:14:26.378458 23813 external_mini_cluster.cc:1428] Setting key 05caf7701e990349b9c6333c020e9e17
Aug 10 18:14:26 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385666, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:26.387130 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.375789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:60163 (local address 127.23.65.126:38855)
0810 18:14:26.375914 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:26.375917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:26.376865 (+ 948us) server_negotiation.cc:407] Connection header received
0810 18:14:26.377750 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:26.377754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:26.377814 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:26.377889 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:26.379191 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.379918 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.381204 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.381420 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.381497 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.383959 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:26.383963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:26.383965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:26.383990 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:26.385319 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:26.385814 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:26.385817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:26.385818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:26.385904 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:26.386213 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:26.386216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:26.386217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:26.386400 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:26.386479 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:26.386910 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 18:14:26.387004 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:14:26.387980 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.376069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:60163)
0810 18:14:26.376710 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:26.376753 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:26.377548 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:26.378099 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:26.378107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:26.378497 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:26.379008 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.379024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.380040 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.380044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.381097 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.381108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.381534 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.381536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.381684 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.382304 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:26.382322 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:26.383816 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:26.385434 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:26.385438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:26.385450 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:26.385706 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:26.385981 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:26.385983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:26.385984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:26.386112 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:26.386479 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:26.386483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:26.386708 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 18:14:26.386856 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 18:14:26.387102 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":103,"threads_started":1}
I20260810 18:14:26.388981 17453 heartbeater.cc:347] Connected to a master server at 127.23.65.126:38855
I20260810 18:14:26.389204 17453 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:26.389667 17453 heartbeater.cc:511] Master 127.23.65.126:38855 requested a full tablet report, sending...
I20260810 18:14:26.390633 16689 ts_manager.cc:194] Registered new tserver with Master: 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889)
I20260810 18:14:26.391223 16689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:60163
I20260810 18:14:26.392307 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:14:26.400830 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.394470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.23.65.126:38855)
0810 18:14:26.394570 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:26.394573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:26.394772 (+ 199us) server_negotiation.cc:407] Connection header received
0810 18:14:26.394867 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:26.394869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:26.394903 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:26.394963 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:26.395621 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.396096 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.396865 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.397057 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.397118 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.398265 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:26.398268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:26.398270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:26.398299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:26.399588 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:26.399913 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:26.399915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:26.399916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:26.399984 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:26.400185 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:26.400187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:26.400188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:26.400388 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:26.400487 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:26.400632 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 18:14:26.400718 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 18:14:26.403417 16689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:14:26.406420 16689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:14:26.420780 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.415232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:36889 (local address 127.0.0.1:48678)
0810 18:14:26.415588 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:26.415602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:26.415730 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:26.416565 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:26.416569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:26.416585 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:26.416733 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.416740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.417703 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.417706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.419013 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.419022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.420212 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.420216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.420324 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.420369 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 18:14:26.420501 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 18:14:26.420624 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":106,"thread_start_us":119,"threads_started":1}
I20260810 18:14:26.420814 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.415522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.23.65.67:36889)
0810 18:14:26.416161 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:26.416174 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:26.416195 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:14:26.416268 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:26.416273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:26.416450 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:26.416583 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:26.416858 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.417575 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.419189 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.420107 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.420256 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.420384 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:26.420553 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 18:14:26.420650 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":159,"threads_started":1}
I20260810 18:14:26.421010 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.414925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.23.65.66:45489)
0810 18:14:26.415248 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:26.415261 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:26.415659 (+ 398us) server_negotiation.cc:407] Connection header received
0810 18:14:26.415935 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:26.415941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:26.416158 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:26.416287 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:26.416800 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.417558 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.419189 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.420096 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.420206 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.420365 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:26.420776 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 18:14:26.420875 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
I20260810 18:14:26.421232 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.414468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:44543 (local address 127.0.0.1:40482)
0810 18:14:26.415391 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:26.415866 (+ 475us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:26.415943 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:26.416414 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:26.416420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:26.416443 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:26.416672 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.416681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.418955 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.418959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.419777 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.419787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.420866 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.420870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.420966 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.420989 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:14:26.421038 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 18:14:26.421105 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":64,"thread_start_us":90,"threads_started":1}
I20260810 18:14:26.421797 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.414543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:45489 (local address 127.0.0.1:51362)
0810 18:14:26.415487 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:26.415559 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:26.415743 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:26.416495 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:26.416498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:26.416516 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:26.416683 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.416691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.417685 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.417689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.419057 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.419066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.420524 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.420528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.420616 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.420639 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:14:26.421592 (+ 953us) client_negotiation.cc:240] Negotiation successful
0810 18:14:26.421673 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":103,"thread_start_us":84,"threads_started":1}
I20260810 18:14:26.422053 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.414630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.23.65.65:44543)
0810 18:14:26.416033 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:26.416039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:26.416060 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:14:26.416123 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:26.416127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:26.416268 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:26.416404 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:26.417709 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.418838 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.419906 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.420726 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.421608 (+ 882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.421763 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:26.421815 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 18:14:26.421912 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":141,"threads_started":1}
I20260810 18:14:26.424173 17345 tablet_service.cc:1467] Processing CreateTablet for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 (DEFAULT_TABLE table=test-table [id=af10b6a97f4e470e8cabc07393a63f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:26.424674 16894 tablet_service.cc:1467] Processing CreateTablet for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 (DEFAULT_TABLE table=test-table [id=af10b6a97f4e470e8cabc07393a63f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:26.425418 17345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:26.425697 16894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:26.425899 17125 tablet_service.cc:1467] Processing CreateTablet for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 (DEFAULT_TABLE table=test-table [id=af10b6a97f4e470e8cabc07393a63f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:26.426740 17125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:26.432832 17479 tablet_bootstrap.cc:492] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: Bootstrap starting.
I20260810 18:14:26.433455 17480 tablet_bootstrap.cc:492] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Bootstrap starting.
I20260810 18:14:26.435341 17480 tablet_bootstrap.cc:654] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:26.435899 17479 tablet_bootstrap.cc:654] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:26.436122 17480 log.cc:824] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:26.436688 17479 log.cc:824] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:26.436825 17481 tablet_bootstrap.cc:492] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: Bootstrap starting.
I20260810 18:14:26.438330 17480 tablet_bootstrap.cc:492] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: No bootstrap required, opened a new log
I20260810 18:14:26.438542 17480 ts_tablet_manager.cc:1397] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:14:26.438696 17479 tablet_bootstrap.cc:492] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: No bootstrap required, opened a new log
I20260810 18:14:26.438894 17479 ts_tablet_manager.cc:1397] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:14:26.439077 17481 tablet_bootstrap.cc:654] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:26.440277 17481 log.cc:824] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:26.442118 17480 raft_consensus.cc:348] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:26.442277 17481 tablet_bootstrap.cc:492] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: No bootstrap required, opened a new log
I20260810 18:14:26.442353 17480 raft_consensus.cc:374] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:26.442436 17480 raft_consensus.cc:729] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7a1a98b3684862af61b46ffcfb27be, State: Initialized, Role: FOLLOWER
I20260810 18:14:26.442459 17481 ts_tablet_manager.cc:1397] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:14:26.442554 17479 raft_consensus.cc:348] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:26.442833 17479 raft_consensus.cc:374] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:26.442934 17479 raft_consensus.cc:729] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c413b3c61ae44cbac0b257c54cacbb0, State: Initialized, Role: FOLLOWER
I20260810 18:14:26.442893 17480 consensus_queue.cc:260] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:26.443370 17479 consensus_queue.cc:260] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:26.443614 17005 heartbeater.cc:503] Master 127.23.65.126:38855 was elected leader, sending a full tablet report...
I20260810 18:14:26.443994 17480 ts_tablet_manager.cc:1428] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 18:14:26.444401 17479 ts_tablet_manager.cc:1428] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:14:26.444825 17231 heartbeater.cc:503] Master 127.23.65.126:38855 was elected leader, sending a full tablet report...
I20260810 18:14:26.445915 17481 raft_consensus.cc:348] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:26.446139 17481 raft_consensus.cc:374] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:26.446172 17481 raft_consensus.cc:729] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634a73f479994917ad451dd54ce7cddf, State: Initialized, Role: FOLLOWER
I20260810 18:14:26.446604 17481 consensus_queue.cc:260] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:26.447275 17453 heartbeater.cc:503] Master 127.23.65.126:38855 was elected leader, sending a full tablet report...
I20260810 18:14:26.447484 17481 ts_tablet_manager.cc:1428] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:14:26.492835 17486 raft_consensus.cc:482] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:14:26.493111 17486 raft_consensus.cc:504] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:26.494410 17486 leader_election.cc:288] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc7a1a98b3684862af61b46ffcfb27be (127.23.65.65:44543), 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889)
I20260810 18:14:26.498265 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.494625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:44543 (local address 127.23.65.66:49547)
0810 18:14:26.494762 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:26.494779 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:26.494872 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:26.495227 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:26.495229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:26.495248 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:26.495351 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.495356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.496123 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.496127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.497062 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.497072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.497841 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.497844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.497950 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.497976 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:14:26.498055 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:14:26.498127 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":22}
I20260810 18:14:26.498263 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.494742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:49547 (local address 127.23.65.65:44543)
0810 18:14:26.494897 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:26.494902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:26.494918 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:14:26.494978 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:26.494981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:26.495031 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:26.495116 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:26.495464 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.496016 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.497189 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.497737 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.497837 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.497879 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:26.498063 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 18:14:26.498133 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:14:26.498564 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.495055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:53203 (local address 127.23.65.67:36889)
0810 18:14:26.495180 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:26.495185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:26.495337 (+ 152us) server_negotiation.cc:407] Connection header received
0810 18:14:26.495443 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:26.495446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:26.495485 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:26.495560 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:26.495815 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.496406 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.497571 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.498028 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.498128 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.498171 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:26.498411 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 18:14:26.498475 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:14:26.498898 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.494928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:36889 (local address 127.23.65.66:53203)
0810 18:14:26.495247 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:26.495260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:26.495357 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:26.495576 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:26.495579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:26.495594 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:26.495710 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.495715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.496532 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.496535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.497450 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:26.497461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.498142 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.498145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:26.498244 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.498267 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:14:26.498690 (+ 423us) client_negotiation.cc:240] Negotiation successful
0810 18:14:26.498766 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":2,"thread_start_us":134,"threads_started":1}
I20260810 18:14:26.498942 16931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe744c7886b4c95ba5b9ebe5bbf7e70" candidate_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7a1a98b3684862af61b46ffcfb27be" is_pre_election: true
I20260810 18:14:26.499246 16931 raft_consensus.cc:2454] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c413b3c61ae44cbac0b257c54cacbb0 in term 0.
I20260810 18:14:26.499495 17376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe744c7886b4c95ba5b9ebe5bbf7e70" candidate_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634a73f479994917ad451dd54ce7cddf" is_pre_election: true
I20260810 18:14:26.499722 17043 leader_election.cc:302] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c413b3c61ae44cbac0b257c54cacbb0, dc7a1a98b3684862af61b46ffcfb27be; no voters:
I20260810 18:14:26.499809 17376 raft_consensus.cc:2454] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c413b3c61ae44cbac0b257c54cacbb0 in term 0.
I20260810 18:14:26.499993 17486 raft_consensus.cc:2781] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:14:26.500074 17486 raft_consensus.cc:482] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:14:26.500141 17486 raft_consensus.cc:3037] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:26.501216 17486 raft_consensus.cc:504] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:26.501525 17486 leader_election.cc:288] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [CANDIDATE]: Term 1 election: Requested vote from peers dc7a1a98b3684862af61b46ffcfb27be (127.23.65.65:44543), 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889)
I20260810 18:14:26.501842 16931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe744c7886b4c95ba5b9ebe5bbf7e70" candidate_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7a1a98b3684862af61b46ffcfb27be"
I20260810 18:14:26.501919 17376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe744c7886b4c95ba5b9ebe5bbf7e70" candidate_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634a73f479994917ad451dd54ce7cddf"
I20260810 18:14:26.501968 16931 raft_consensus.cc:3037] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:26.502035 17376 raft_consensus.cc:3037] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:26.502992 16931 raft_consensus.cc:2454] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c413b3c61ae44cbac0b257c54cacbb0 in term 1.
I20260810 18:14:26.503015 17376 raft_consensus.cc:2454] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c413b3c61ae44cbac0b257c54cacbb0 in term 1.
I20260810 18:14:26.503270 17043 leader_election.cc:302] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c413b3c61ae44cbac0b257c54cacbb0, dc7a1a98b3684862af61b46ffcfb27be; no voters:
I20260810 18:14:26.503439 17486 raft_consensus.cc:2781] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:14:26.503674 17486 raft_consensus.cc:686] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 LEADER]: Becoming Leader. State: Replica: 6c413b3c61ae44cbac0b257c54cacbb0, State: Running, Role: LEADER
I20260810 18:14:26.504005 17486 consensus_queue.cc:237] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:26.507014 16688 catalog_manager.cc:5495] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } health_report { overall_health: UNKNOWN } } }
I20260810 18:14:26.519979 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.516847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.23.65.66:45489)
0810 18:14:26.516945 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:26.516949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:26.516975 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:14:26.517103 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:26.517105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:26.517153 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:26.517234 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:14:26.517426 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.517893 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.518668 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:26.518878 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:26.518951 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:26.519163 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:14:26.519661 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:14:26.519750 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:26.519813 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 18:14:26.519860 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 18:14:26.525507 17376 raft_consensus.cc:1261] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [term 1 FOLLOWER]: Refusing update from remote peer 6c413b3c61ae44cbac0b257c54cacbb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:26.525508 16931 raft_consensus.cc:1261] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Refusing update from remote peer 6c413b3c61ae44cbac0b257c54cacbb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:26.526156 17493 consensus_queue.cc:1035] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:26.526325 17486 consensus_queue.cc:1035] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:26.534448 17500 mvcc.cc:205] Tried to move back new op lower bound from 7317035690081865728 to 7317035690002800640. Current Snapshot: MvccSnapshot[applied={T|T < 7317035690081865728}]
I20260810 18:14:26.536953 17499 mvcc.cc:205] Tried to move back new op lower bound from 7317035690081865728 to 7317035690002800640. Current Snapshot: MvccSnapshot[applied={T|T < 7317035690081865728 or (T in {7317035690081865728})}]
I20260810 18:14:26.540999 16688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:14:26.541150 16688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:14:26.543465 16688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Sending DeleteTablet for 3 replicas of tablet ffe744c7886b4c95ba5b9ebe5bbf7e70
I20260810 18:14:26.544283 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 16791
I20260810 18:14:26.544222 17125 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:26 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:51362
I20260810 18:14:26.544418 16894 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:26 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:40482
I20260810 18:14:26.544697 17345 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:26 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:48678
I20260810 18:14:26.549803 17511 tablet_replica.cc:330] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: stopping tablet replica
I20260810 18:14:26.550140 17511 raft_consensus.cc:2229] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:14:26.550465 17511 raft_consensus.cc:2258] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:26.551520 17510 tablet_replica.cc:330] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: stopping tablet replica
I20260810 18:14:26.551833 17510 raft_consensus.cc:2229] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:14:26.552166 17510 raft_consensus.cc:2258] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:26.555666 17510 ts_tablet_manager.cc:1905] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:26.556483 17511 ts_tablet_manager.cc:1905] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:26.558455 17510 ts_tablet_manager.cc:1918] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:26.558624 17510 log.cc:1196] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/ffe744c7886b4c95ba5b9ebe5bbf7e70
I20260810 18:14:26.559015 17510 ts_tablet_manager.cc:1939] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 6c413b3c61ae44cbac0b257c54cacbb0: Deleting consensus metadata
I20260810 18:14:26.559130 17511 ts_tablet_manager.cc:1918] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:26.559314 17511 log.cc:1196] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/ffe744c7886b4c95ba5b9ebe5bbf7e70
I20260810 18:14:26.559741 17511 ts_tablet_manager.cc:1939] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P 634a73f479994917ad451dd54ce7cddf: Deleting consensus metadata
I20260810 18:14:26.560636 16662 catalog_manager.cc:4841] TS 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489): tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 (table test-table [id=af10b6a97f4e470e8cabc07393a63f1e]) successfully deleted
W20260810 18:14:26.564083 16664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:44543 (error 108)
I20260810 18:14:26.564679 16664 catalog_manager.cc:4841] TS 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889): tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 (table test-table [id=af10b6a97f4e470e8cabc07393a63f1e]) successfully deleted
I20260810 18:14:26.564987 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 17012
I20260810 18:14:26.565227 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:26.564952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:44543 (local address 127.0.0.1:40494)
0810 18:14:26.565077 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:26.565145 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:44543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 18:14:26.576576 16664 catalog_manager.cc:4584] TS dc7a1a98b3684862af61b46ffcfb27be (127.23.65.65:44543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70: Network error: Client connection negotiation failed: client connection to 127.23.65.65:44543: connect: Connection refused (error 111)
I20260810 18:14:26.583071 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 17238
I20260810 18:14:26.595726 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 16638
I20260810 18:14:46.610970 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38855
--webserver_interface=127.23.65.126
--webserver_port=36641
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:14:46.724972 17514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:46.725236 17514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:46.725276 17514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:46.729267 17514 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 18:14:46.729321 17514 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 18:14:46.729336 17514 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 18:14:46.729349 17514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:14:46.729365 17514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:46.729385 17514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:46.729416 17514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:14:46.729442 17514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:14:46.734671 17514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=36641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:46.735778 17514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:46.736816 17514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:46.750720 17524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:46.750721 17525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:46.750808 17514 server_base.cc:1034] running on GCE node
W20260810 18:14:46.750721 17530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:46.751677 17514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:46.752866 17514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:46.754033 17514 hybrid_clock.cc:648] HybridClock initialized: now 1786385686754016 us; error 34 us; skew 500 ppm
Aug 10 18:14:46 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:46.759461 17514 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:14:46.760452 17514 webserver.cc:460] Webserver started at http://127.23.65.126:36641/ using document root <none> and password file <none>
I20260810 18:14:46.761026 17514 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:46.761111 17514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:46.765045 17514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:14:46.767177 17540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:46.768384 17514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:46.768505 17514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "6582f75204bf4effaf30f27f8948c008"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "844a81dec5f577e1751cb0585e5193f0"
server_key_iv: "f9e517d700591b68d0b84a2d3c93889e"
server_key_version: "encryptionkey@0"
I20260810 18:14:46.769094 17514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:46.780495 17514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:46.783488 17514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:46.783691 17514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:46.790314 17514 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:38855
I20260810 18:14:46.790563 17615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:38855 every 8 connection(s)
I20260810 18:14:46.791392 17514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:14:46.795899 17616 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:14:46.797580 17616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Bootstrap starting.
I20260810 18:14:46.798868 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 17514
I20260810 18:14:46.799289 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:44543
--local_ip_for_outbound_sockets=127.23.65.65
--tserver_master_addrs=127.23.65.126:38855
--webserver_port=42039
--webserver_interface=127.23.65.65
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:14:46.815550 17616 log.cc:824] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:46.824561 17616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:14:46.824929 17616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Bootstrap complete.
I20260810 18:14:46.829015 17616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:14:46.829423 17616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6582f75204bf4effaf30f27f8948c008, State: Initialized, Role: FOLLOWER
I20260810 18:14:46.829900 17616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:14:46.830025 17616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:14:46.830101 17616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:14:46.830194 17616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:14:46.832114 17616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:14:46.832454 17616 leader_election.cc:302] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6582f75204bf4effaf30f27f8948c008; no voters:
I20260810 18:14:46.832762 17616 leader_election.cc:288] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:14:46.832901 17620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:14:46.833127 17620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 2 LEADER]: Becoming Leader. State: Replica: 6582f75204bf4effaf30f27f8948c008, State: Running, Role: LEADER
I20260810 18:14:46.833491 17620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:14:46.834025 17616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:14:46.834937 17624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6582f75204bf4effaf30f27f8948c008. Latest consensus state: current_term: 2 leader_uuid: "6582f75204bf4effaf30f27f8948c008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } } }
I20260810 18:14:46.835071 17624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:46.835461 17642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:14:46.835506 17621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6582f75204bf4effaf30f27f8948c008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } } }
I20260810 18:14:46.835624 17621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: This master's current role is: LEADER
I20260810 18:14:46.840250 17642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:14:46.840889 17642 catalog_manager.cc:1259] Loaded cluster ID: bfa272cfd09047c3bf7244ce6bf547a5
I20260810 18:14:46.841213 17642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:14:46.842945 17642 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:14:46.843482 17642 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Loaded TSK: 0
I20260810 18:14:46.851904 17642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Generated new TSK 1
I20260810 18:14:46.852142 17642 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:14:46.927173 17618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:46.927435 17618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:46.927496 17618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:46.931447 17618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:46.931504 17618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:46.931602 17618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:14:46.937026 17618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:44543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=42039
--tserver_master_addrs=127.23.65.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:46.938006 17618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:46.939002 17618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:46.948626 17667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:46.948792 17618 server_base.cc:1034] running on GCE node
W20260810 18:14:46.948611 17670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:46.948611 17668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:46.949373 17618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:46.950001 17618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:46.951175 17618 hybrid_clock.cc:648] HybridClock initialized: now 1786385686951154 us; error 41 us; skew 500 ppm
Aug 10 18:14:46 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:46.954001 17618 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:14:46.954886 17618 webserver.cc:460] Webserver started at http://127.23.65.65:42039/ using document root <none> and password file <none>
I20260810 18:14:46.955413 17618 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:46.955487 17618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:46.959424 17618 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:14:46.961706 17679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:46.963037 17618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:14:46.963155 17618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "dc7a1a98b3684862af61b46ffcfb27be"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "05ddf81bd24f310428fc3a522469e8ae"
server_key_iv: "b79c07ce88e837db9f774996a168912c"
server_key_version: "encryptionkey@0"
I20260810 18:14:46.963742 17618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:46.976227 17618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:46.979211 17618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:46.979389 17618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:46.980038 17618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:46.981141 17688 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:14:46.985210 17618 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:14:46.985270 17618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260810 18:14:46.985320 17618 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:14:46.986948 17618 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:14:46.987005 17618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 18:14:46.987298 17688 tablet_bootstrap.cc:492] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Bootstrap starting.
I20260810 18:14:46.997848 17618 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:44543
I20260810 18:14:46.997901 17861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:44543 every 8 connection(s)
I20260810 18:14:46.999099 17618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:14:47.006948 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 17618
I20260810 18:14:47.007515 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:45489
--local_ip_for_outbound_sockets=127.23.65.66
--tserver_master_addrs=127.23.65.126:38855
--webserver_port=36445
--webserver_interface=127.23.65.66
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:14:47.008705 17688 log.cc:824] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:14:47 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:47.017233 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.001240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:39071 (local address 127.23.65.126:38855)
0810 18:14:47.001598 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.001615 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.002037 (+ 422us) server_negotiation.cc:407] Connection header received
0810 18:14:47.003130 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.003140 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.003441 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.003774 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:47.004957 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.005941 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.007322 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.008033 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.008209 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.011235 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:47.011242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:47.011258 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:47.011295 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:47.013433 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.014090 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.014097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.014103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.014262 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.014671 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.014674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.014675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.015153 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:47.015283 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.016042 (+ 759us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.016352 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
I20260810 18:14:47.017284 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.001322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:39071)
0810 18:14:47.001866 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:47.001910 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:47.002706 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:47.003870 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:47.003877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:47.004297 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:47.004767 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.004795 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.006100 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.006103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.007173 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.007185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.008167 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.008171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.008393 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.009055 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:47.009077 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:47.011013 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:47.013567 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:47.013574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:47.013589 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:47.013961 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:47.014379 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:47.014383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:47.014385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:47.014573 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:47.015293 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:47.015300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:47.015660 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 18:14:47.015887 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 18:14:47.016257 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":177,"threads_started":1}
I20260810 18:14:47.017380 17688 tablet_bootstrap.cc:492] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:14:47.017812 17688 tablet_bootstrap.cc:492] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Bootstrap complete.
I20260810 18:14:47.018903 17688 ts_tablet_manager.cc:1397] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.011s
I20260810 18:14:47.020282 17862 heartbeater.cc:347] Connected to a master server at 127.23.65.126:38855
I20260810 18:14:47.020529 17862 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:47.021099 17862 heartbeater.cc:511] Master 127.23.65.126:38855 requested a full tablet report, sending...
I20260810 18:14:47.021576 17688 raft_consensus.cc:348] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:47.021991 17688 raft_consensus.cc:729] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7a1a98b3684862af61b46ffcfb27be, State: Initialized, Role: FOLLOWER
I20260810 18:14:47.022491 17688 consensus_queue.cc:260] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } } peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } }
I20260810 18:14:47.023607 17688 ts_tablet_manager.cc:1428] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:14:47.023927 17565 ts_manager.cc:194] Registered new tserver with Master: dc7a1a98b3684862af61b46ffcfb27be (127.23.65.65:44543)
I20260810 18:14:47.027464 17565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:39071
I20260810 18:14:47.030236 17862 heartbeater.cc:503] Master 127.23.65.126:38855 was elected leader, sending a full tablet report...
Aug 10 18:14:47 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.65@KRBTEST.COM
I20260810 18:14:47.036737 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.026723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.23.65.65:44543)
0810 18:14:47.027432 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.027448 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.027472 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:14:47.027582 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.027588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.027795 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.027922 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:47.028663 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.029384 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.030228 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.030646 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.030738 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.033147 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:47.033157 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:47.033174 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:47.033199 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:47.034710 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.035232 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.035238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.035240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.035351 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.035761 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.035765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.035766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.036022 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:47.036124 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.036470 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.036608 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":103,"threads_started":1}
I20260810 18:14:47.036793 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.026657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:44543 (local address 127.0.0.1:41466)
0810 18:14:47.027096 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:47.027143 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:47.027264 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:47.027948 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:47.027953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:47.028412 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:47.028549 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.028557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.029494 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.029497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.030113 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.030123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.030753 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.030757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.030870 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.031450 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:47.031465 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:47.033018 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:47.034810 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:47.034820 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:47.034826 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:47.035107 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:47.035513 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:47.035516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:47.035519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:47.035655 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:47.036128 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:47.036135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:47.036265 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 18:14:47.036413 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 18:14:47.036611 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":133,"threads_started":1}
I20260810 18:14:47.037770 17754 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:26 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:41466
I20260810 18:14:47.038762 17874 tablet_replica.cc:330] stopping tablet replica
I20260810 18:14:47.039055 17874 raft_consensus.cc:2229] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:14:47.039291 17874 raft_consensus.cc:2258] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:47.040277 17874 ts_tablet_manager.cc:1905] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:47.043983 17874 ts_tablet_manager.cc:1918] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:47.044070 17874 log.cc:1196] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal/wals/ffe744c7886b4c95ba5b9ebe5bbf7e70
I20260810 18:14:47.044301 17874 ts_tablet_manager.cc:1939] T ffe744c7886b4c95ba5b9ebe5bbf7e70 P dc7a1a98b3684862af61b46ffcfb27be: Deleting consensus metadata
I20260810 18:14:47.046202 17545 catalog_manager.cc:4841] TS dc7a1a98b3684862af61b46ffcfb27be (127.23.65.65:44543): tablet ffe744c7886b4c95ba5b9ebe5bbf7e70 (table test-table [id=af10b6a97f4e470e8cabc07393a63f1e]) successfully deleted
W20260810 18:14:47.127683 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:47.127945 17870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:47.128005 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:47.132043 17870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:47.132112 17870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:47.132192 17870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:14:47.137689 17870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:45489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=36445
--tserver_master_addrs=127.23.65.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:47.138761 17870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:47.139842 17870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:47.149382 17887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:47.149410 17881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:47.149639 17870 server_base.cc:1034] running on GCE node
W20260810 18:14:47.149382 17882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:47.150197 17870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:47.150862 17870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:47.152022 17870 hybrid_clock.cc:648] HybridClock initialized: now 1786385687152002 us; error 31 us; skew 500 ppm
Aug 10 18:14:47 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385687, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:47.154834 17870 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:14:47.155783 17870 webserver.cc:460] Webserver started at http://127.23.65.66:36445/ using document root <none> and password file <none>
I20260810 18:14:47.156344 17870 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:47.156416 17870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:47.160327 17870 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 18:14:47.162470 17904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:47.163774 17870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:14:47.163877 17870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "6c413b3c61ae44cbac0b257c54cacbb0"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "46c73ba6f351beb7482b444e218ad0a3"
server_key_iv: "34bb6cb029e2a838e176f17b759924ee"
server_key_version: "encryptionkey@0"
I20260810 18:14:47.164438 17870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:47.179952 17870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:47.183008 17870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:47.183164 17870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:47.183734 17870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:47.184543 17870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:47.184592 17870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:47.184631 17870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:47.184651 17870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:47.193635 17870 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:45489
I20260810 18:14:47.193691 18080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:45489 every 8 connection(s)
I20260810 18:14:47.194656 17870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:14:47.195463 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 17870
I20260810 18:14:47.195928 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:36889
--local_ip_for_outbound_sockets=127.23.65.67
--tserver_master_addrs=127.23.65.126:38855
--webserver_port=38983
--webserver_interface=127.23.65.67
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:14:47 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385687, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:47.208982 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.196524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:50181 (local address 127.23.65.126:38855)
0810 18:14:47.196656 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.196659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.197712 (+ 1053us) server_negotiation.cc:407] Connection header received
0810 18:14:47.199040 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.199044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.199103 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.199182 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:47.200445 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.201040 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.202187 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.202432 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.202524 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.205441 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:47.205445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:47.205447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:47.205475 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:47.206806 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.207432 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.207434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.207436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.207508 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.207895 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.207898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.207899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.208078 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:47.208153 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.208735 (+ 582us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.208851 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:14:47.210115 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.197041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:50181)
0810 18:14:47.197561 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:47.197606 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:47.198829 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:47.199383 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:47.199390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:47.199806 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:47.200294 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.200309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.201169 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.201172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.202067 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.202077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.202579 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.202584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.202745 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.203399 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:47.203416 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:47.205247 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:47.206929 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:47.206936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:47.206953 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:47.207342 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:47.207614 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:47.207617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:47.207619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:47.207814 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:47.208166 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:47.208173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:47.208504 (+ 331us) client_negotiation.cc:769] Sending connection context
0810 18:14:47.208709 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 18:14:47.209054 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":145,"threads_started":1}
I20260810 18:14:47.211315 18082 heartbeater.cc:347] Connected to a master server at 127.23.65.126:38855
I20260810 18:14:47.211572 18082 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:47.212091 18082 heartbeater.cc:511] Master 127.23.65.126:38855 requested a full tablet report, sending...
I20260810 18:14:47.213181 17565 ts_manager.cc:194] Registered new tserver with Master: 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489)
I20260810 18:14:47.213748 17565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:50181
W20260810 18:14:47.313848 18088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:14:47.314116 18088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:14:47.314173 18088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:14:47.318114 18088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:14:47.318173 18088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:14:47.318252 18088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:14:47.323319 18088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:36889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=38983
--tserver_master_addrs=127.23.65.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:14:47.324342 18088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:14:47.325338 18088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:14:47.334645 18099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:47.334679 18088 server_base.cc:1034] running on GCE node
W20260810 18:14:47.334638 18106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:14:47.334916 18100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:14:47.335340 18088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:14:47.336006 18088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:14:47.337210 18088 hybrid_clock.cc:648] HybridClock initialized: now 1786385687337191 us; error 30 us; skew 500 ppm
Aug 10 18:14:47 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385687, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:14:47.339841 18088 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:14:47.340706 18088 webserver.cc:460] Webserver started at http://127.23.65.67:38983/ using document root <none> and password file <none>
I20260810 18:14:47.341233 18088 fs_manager.cc:359] Metadata directory not provided
I20260810 18:14:47.341343 18088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:14:47.345011 18088 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:14:47.347062 18114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:47.348274 18088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:14:47.348394 18088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "634a73f479994917ad451dd54ce7cddf"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "2fe0dd5a34b3296393ec19162824b43d"
server_key_iv: "a0104bcb812bd3f089f6ac3e47a3aaa7"
server_key_version: "encryptionkey@0"
I20260810 18:14:47.348913 18088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:14:47.364446 18088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:14:47.367302 18088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:14:47.367509 18088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:14:47.368146 18088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:14:47.368999 18088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:14:47.369050 18088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:47.369103 18088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:14:47.369130 18088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:14:47.378057 18088 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:36889
I20260810 18:14:47.378084 18281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:36889 every 8 connection(s)
I20260810 18:14:47.379060 18088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:14:47.383491 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 18088
Aug 10 18:14:47 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385687, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:14:47.392643 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.384204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.23.65.126:38855)
0810 18:14:47.384430 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.384434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.384588 (+ 154us) server_negotiation.cc:407] Connection header received
0810 18:14:47.384719 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.384722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.384764 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.384838 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:47.385681 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.386472 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.387555 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.387842 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.387937 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.389178 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:47.389182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:47.389183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:47.389207 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:47.390974 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.391405 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.391407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.391409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.391485 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.391767 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.391769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.391771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.391960 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:47.392152 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.392300 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.392453 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"spinlock_wait_cycles":9088,"thread_start_us":71,"threads_started":1}
I20260810 18:14:47.393260 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.380844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:55965 (local address 127.23.65.126:38855)
0810 18:14:47.380972 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.380976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.382024 (+ 1048us) server_negotiation.cc:407] Connection header received
0810 18:14:47.382900 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.382903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.382958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.383035 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:47.384524 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.385325 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.386833 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.387180 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.387281 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.389689 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:47.389693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:47.389695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:47.389736 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:47.391148 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.391681 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.391683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.391684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.391758 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.392094 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.392096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.392098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.392309 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:47.392429 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.392974 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.393084 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:14:47.394254 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.381233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:55965)
0810 18:14:47.381880 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:47.381923 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:47.382696 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:47.383231 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:47.383241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:47.383771 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:14:47.384361 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.384382 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.385475 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.385478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.386714 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.386724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.387306 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.387309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.387448 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.388054 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:14:47.388073 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:14:47.389499 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:14:47.391247 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:47.391253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:47.391269 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:47.391597 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:47.391850 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:14:47.391852 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:14:47.391854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:14:47.391989 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:14:47.392422 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:14:47.392429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:14:47.392690 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 18:14:47.392905 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 18:14:47.393381 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":119,"threads_started":1}
I20260810 18:14:47.395356 18283 heartbeater.cc:347] Connected to a master server at 127.23.65.126:38855
I20260810 18:14:47.395618 18283 heartbeater.cc:464] Registering TS with master...
I20260810 18:14:47.396116 18283 heartbeater.cc:511] Master 127.23.65.126:38855 requested a full tablet report, sending...
I20260810 18:14:47.397149 17565 ts_manager.cc:194] Registered new tserver with Master: 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889)
I20260810 18:14:47.397686 17565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:55965
I20260810 18:14:47.406363 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:14:47.410288 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.407062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.23.65.126:38855)
0810 18:14:47.407170 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.407173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.407408 (+ 235us) server_negotiation.cc:407] Connection header received
0810 18:14:47.407576 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.407579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.407696 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.407792 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:14:47.407987 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.408493 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.409285 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.409485 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.409551 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.409820 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:14:47.409963 (+ 143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 18:14:47.410169 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59630: token expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260810 18:14:47.410357 17867 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59630: token expired
W20260810 18:14:47.410668 17459 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.65.126:38855)
I20260810 18:14:47.417440 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.411032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.23.65.126:38855)
0810 18:14:47.411132 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.411135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.411146 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:14:47.411232 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.411234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.411278 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.411335 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:14:47.412019 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.412520 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.413240 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.413437 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.413501 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.414641 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:14:47.414644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:14:47.414646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:14:47.414669 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:14:47.416134 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.416528 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.416531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.416532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.416606 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:14:47.416803 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:14:47.416805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:14:47.416806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:14:47.416973 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:14:47.417070 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.417220 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.417315 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 18:14:47.419978 17565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:14:47.423345 17565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:14:47.432147 17754 tablet_service.cc:1467] Processing CreateTablet for tablet c13ad9a9d67b42ebaca5b7a6120614da (DEFAULT_TABLE table=test-table [id=e2743e49859b4750a4c50345b7fdb663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:47.433038 17754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c13ad9a9d67b42ebaca5b7a6120614da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:47.436450 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.431225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.23.65.66:45489)
0810 18:14:47.431710 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.431721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.431743 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:14:47.431842 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.431853 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.432094 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.432270 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:47.432670 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.433772 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.435134 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.435831 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.435933 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.436097 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.436203 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.436301 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":104,"threads_started":1}
I20260810 18:14:47.436440 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.430861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:45489 (local address 127.0.0.1:55504)
0810 18:14:47.431216 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:47.431232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:47.431318 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:47.432269 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:47.432272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:47.432295 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:47.432482 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.432500 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.433906 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.433909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.435040 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.435048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.435945 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.435948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.436049 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.436098 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 18:14:47.436172 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 18:14:47.436261 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260810 18:14:47.436697 17688 tablet_bootstrap.cc:492] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Bootstrap starting.
I20260810 18:14:47.436772 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.431052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:36889 (local address 127.0.0.1:60566)
0810 18:14:47.431343 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:47.431355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:47.431445 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:47.432530 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:47.432532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:47.432549 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:47.432640 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.432646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.433631 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.433634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.434951 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.434963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.436376 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.436379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.436471 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.436495 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:14:47.436565 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:14:47.436636 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":143,"threads_started":1}
I20260810 18:14:47.437371 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.431518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.23.65.67:36889)
0810 18:14:47.431952 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.431963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.431985 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:14:47.432052 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.432056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.432216 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.432357 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:47.432772 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.433518 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.435089 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.436038 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.436153 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.436305 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.436966 (+ 661us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.437073 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":183,"threads_started":1}
I20260810 18:14:47.437620 17688 tablet_bootstrap.cc:654] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:47.439340 17688 tablet_bootstrap.cc:492] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: No bootstrap required, opened a new log
I20260810 18:14:47.439445 17688 ts_tablet_manager.cc:1397] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:14:47.439968 17688 raft_consensus.cc:348] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:14:47.440109 17688 raft_consensus.cc:374] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:47.440150 17688 raft_consensus.cc:729] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7a1a98b3684862af61b46ffcfb27be, State: Initialized, Role: FOLLOWER
I20260810 18:14:47.440373 17688 consensus_queue.cc:260] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:14:47.440786 17688 ts_tablet_manager.cc:1428] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 18:14:47.440861 18181 tablet_service.cc:1467] Processing CreateTablet for tablet c13ad9a9d67b42ebaca5b7a6120614da (DEFAULT_TABLE table=test-table [id=e2743e49859b4750a4c50345b7fdb663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:47.440860 17981 tablet_service.cc:1467] Processing CreateTablet for tablet c13ad9a9d67b42ebaca5b7a6120614da (DEFAULT_TABLE table=test-table [id=e2743e49859b4750a4c50345b7fdb663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:14:47.442039 17981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c13ad9a9d67b42ebaca5b7a6120614da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:47.442044 18181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c13ad9a9d67b42ebaca5b7a6120614da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:14:47.449275 18300 tablet_bootstrap.cc:492] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: Bootstrap starting.
I20260810 18:14:47.449366 18302 tablet_bootstrap.cc:492] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: Bootstrap starting.
I20260810 18:14:47.452183 18302 tablet_bootstrap.cc:654] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:47.452210 18300 tablet_bootstrap.cc:654] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: Neither blocks nor log segments found. Creating new log.
I20260810 18:14:47.452976 18302 log.cc:824] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:47.453013 18300 log.cc:824] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: Log is configured to *not* fsync() on all Append() calls
I20260810 18:14:47.454873 18302 tablet_bootstrap.cc:492] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: No bootstrap required, opened a new log
I20260810 18:14:47.454888 18300 tablet_bootstrap.cc:492] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: No bootstrap required, opened a new log
I20260810 18:14:47.455055 18300 ts_tablet_manager.cc:1397] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:14:47.455054 18302 ts_tablet_manager.cc:1397] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:14:47.458233 18302 raft_consensus.cc:348] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:14:47.458232 18300 raft_consensus.cc:348] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:14:47.458482 18300 raft_consensus.cc:374] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:47.458487 18302 raft_consensus.cc:374] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:14:47.458518 18300 raft_consensus.cc:729] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634a73f479994917ad451dd54ce7cddf, State: Initialized, Role: FOLLOWER
I20260810 18:14:47.458539 18302 raft_consensus.cc:729] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c413b3c61ae44cbac0b257c54cacbb0, State: Initialized, Role: FOLLOWER
I20260810 18:14:47.458945 18300 consensus_queue.cc:260] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:14:47.458943 18302 consensus_queue.cc:260] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:14:47.459698 18283 heartbeater.cc:503] Master 127.23.65.126:38855 was elected leader, sending a full tablet report...
I20260810 18:14:47.459941 18082 heartbeater.cc:503] Master 127.23.65.126:38855 was elected leader, sending a full tablet report...
I20260810 18:14:47.459951 18302 ts_tablet_manager.cc:1428] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:14:47.460813 18300 ts_tablet_manager.cc:1428] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260810 18:14:47.500439 17866 tablet.cc:2369] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:14:47.598819 18308 raft_consensus.cc:482] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:14:47.599041 18308 raft_consensus.cc:504] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:14:47.600278 18308 leader_election.cc:288] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889), dc7a1a98b3684862af61b46ffcfb27be (127.23.65.65:44543)
I20260810 18:14:47.605165 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.600655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:35085 (local address 127.23.65.67:36889)
0810 18:14:47.600782 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.600785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.600805 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:14:47.601000 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.601003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.601061 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.601149 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:47.601505 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.602274 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.603886 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.604507 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.604640 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.604696 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.604958 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.605036 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:14:47.605168 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.600989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:55411 (local address 127.23.65.65:44543)
0810 18:14:47.601277 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.601281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.601295 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:14:47.601397 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.601399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.601457 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.601572 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:14:47.601809 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.602503 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.603900 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.604557 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.604663 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.604834 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.604950 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.605032 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260810 18:14:47.605180 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.600576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:36889 (local address 127.23.65.66:35085)
0810 18:14:47.600748 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:47.600764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:47.600856 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:47.601227 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:47.601231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:47.601253 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:47.601385 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.601391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.602425 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.602428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.603756 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.603769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.604641 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.604675 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.604846 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.604864 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:14:47.604926 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:14:47.605019 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":61}
I20260810 18:14:47.605189 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.600859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:44543 (local address 127.23.65.66:55411)
0810 18:14:47.601230 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:47.601243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:14:47.601317 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:14:47.601585 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:14:47.601587 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:14:47.601603 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:14:47.601702 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.601706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.602627 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.602630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.603799 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:14:47.603806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.604655 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.604658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:14:47.604800 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.604846 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 18:14:47.604927 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 18:14:47.605013 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":32,"thread_start_us":136,"threads_started":1}
I20260810 18:14:47.605923 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13ad9a9d67b42ebaca5b7a6120614da" candidate_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634a73f479994917ad451dd54ce7cddf" is_pre_election: true
I20260810 18:14:47.605974 17791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13ad9a9d67b42ebaca5b7a6120614da" candidate_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7a1a98b3684862af61b46ffcfb27be" is_pre_election: true
I20260810 18:14:47.606264 18211 raft_consensus.cc:2454] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c413b3c61ae44cbac0b257c54cacbb0 in term 0.
I20260810 18:14:47.606264 17791 raft_consensus.cc:2454] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c413b3c61ae44cbac0b257c54cacbb0 in term 0.
I20260810 18:14:47.606758 17907 leader_election.cc:302] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634a73f479994917ad451dd54ce7cddf, 6c413b3c61ae44cbac0b257c54cacbb0; no voters:
I20260810 18:14:47.607015 18308 raft_consensus.cc:2781] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:14:47.607115 18308 raft_consensus.cc:482] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:14:47.607168 18308 raft_consensus.cc:3037] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:47.608310 18308 raft_consensus.cc:504] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:14:47.608647 18308 leader_election.cc:288] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [CANDIDATE]: Term 1 election: Requested vote from peers 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889), dc7a1a98b3684862af61b46ffcfb27be (127.23.65.65:44543)
I20260810 18:14:47.609066 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13ad9a9d67b42ebaca5b7a6120614da" candidate_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634a73f479994917ad451dd54ce7cddf"
I20260810 18:14:47.609066 17791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13ad9a9d67b42ebaca5b7a6120614da" candidate_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7a1a98b3684862af61b46ffcfb27be"
I20260810 18:14:47.609202 17791 raft_consensus.cc:3037] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:47.609203 18211 raft_consensus.cc:3037] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:14:47.610469 18211 raft_consensus.cc:2454] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c413b3c61ae44cbac0b257c54cacbb0 in term 1.
I20260810 18:14:47.610469 17791 raft_consensus.cc:2454] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c413b3c61ae44cbac0b257c54cacbb0 in term 1.
I20260810 18:14:47.610764 17907 leader_election.cc:302] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c413b3c61ae44cbac0b257c54cacbb0, dc7a1a98b3684862af61b46ffcfb27be; no voters:
I20260810 18:14:47.610918 18308 raft_consensus.cc:2781] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:14:47.611132 18308 raft_consensus.cc:686] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 LEADER]: Becoming Leader. State: Replica: 6c413b3c61ae44cbac0b257c54cacbb0, State: Running, Role: LEADER
I20260810 18:14:47.611451 18308 consensus_queue.cc:237] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:14:47.614449 17565 catalog_manager.cc:5495] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } health_report { overall_health: UNKNOWN } } }
I20260810 18:14:47.628844 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.625581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.23.65.66:45489)
0810 18:14:47.625710 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:14:47.625713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:14:47.625726 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:14:47.625877 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:14:47.625879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:14:47.625931 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:14:47.626013 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:14:47.626228 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.626714 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.627463 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:14:47.627727 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:14:47.627827 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:14:47.628097 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:14:47.628496 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:14:47.628582 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:14:47.628632 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 18:14:47.628680 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 18:14:47.630003 18284 tablet.cc:2369] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:14:47.634550 18211 raft_consensus.cc:1261] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [term 1 FOLLOWER]: Refusing update from remote peer 6c413b3c61ae44cbac0b257c54cacbb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:47.634577 17791 raft_consensus.cc:1261] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Refusing update from remote peer 6c413b3c61ae44cbac0b257c54cacbb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:14:47.635210 18308 consensus_queue.cc:1035] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:47.635418 18311 consensus_queue.cc:1035] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:14:47.640259 18318 mvcc.cc:205] Tried to move back new op lower bound from 7317035776543879168 to 7317035776459104256. Current Snapshot: MvccSnapshot[applied={T|T < 7317035776543879168}]
I20260810 18:14:47.644248 18320 mvcc.cc:205] Tried to move back new op lower bound from 7317035776543879168 to 7317035776459104256. Current Snapshot: MvccSnapshot[applied={T|T < 7317035776543879168}]
I20260810 18:14:47.649483 17565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:14:47.649643 17565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:14:47.651942 17565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Sending DeleteTablet for 3 replicas of tablet c13ad9a9d67b42ebaca5b7a6120614da
I20260810 18:14:47.652571 17981 tablet_service.cc:1514] Processing DeleteTablet for tablet c13ad9a9d67b42ebaca5b7a6120614da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:47 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:55504
I20260810 18:14:47.652724 17754 tablet_service.cc:1514] Processing DeleteTablet for tablet c13ad9a9d67b42ebaca5b7a6120614da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:47 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:41466
I20260810 18:14:47.652920 18181 tablet_service.cc:1514] Processing DeleteTablet for tablet c13ad9a9d67b42ebaca5b7a6120614da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:14:47 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:60566
I20260810 18:14:47.653410 18330 tablet_replica.cc:330] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: stopping tablet replica
I20260810 18:14:47.653570 18329 tablet_replica.cc:330] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: stopping tablet replica
I20260810 18:14:47.653612 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 17618
I20260810 18:14:47.661283 18328 tablet_replica.cc:330] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: stopping tablet replica
I20260810 18:14:47.661667 18328 raft_consensus.cc:2229] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:14:47.662902 18328 raft_consensus.cc:2258] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:47.665530 18328 ts_tablet_manager.cc:1905] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:47.668617 18328 ts_tablet_manager.cc:1918] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:47.668813 18328 log.cc:1196] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/c13ad9a9d67b42ebaca5b7a6120614da
I20260810 18:14:47.669234 18328 ts_tablet_manager.cc:1939] T c13ad9a9d67b42ebaca5b7a6120614da P 6c413b3c61ae44cbac0b257c54cacbb0: Deleting consensus metadata
I20260810 18:14:47.672283 17541 catalog_manager.cc:4841] TS 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489): tablet c13ad9a9d67b42ebaca5b7a6120614da (table test-table [id=e2743e49859b4750a4c50345b7fdb663]) successfully deleted
I20260810 18:14:47.673800 18330 raft_consensus.cc:2229] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:14:47.674190 18330 raft_consensus.cc:2258] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:14:47.675998 18330 ts_tablet_manager.cc:1905] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:14:47.678786 18330 ts_tablet_manager.cc:1918] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:14:47.678891 18330 log.cc:1196] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/c13ad9a9d67b42ebaca5b7a6120614da
I20260810 18:14:47.679209 18330 ts_tablet_manager.cc:1939] T c13ad9a9d67b42ebaca5b7a6120614da P 634a73f479994917ad451dd54ce7cddf: Deleting consensus metadata
I20260810 18:14:47.680576 17545 catalog_manager.cc:4841] TS 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889): tablet c13ad9a9d67b42ebaca5b7a6120614da (table test-table [id=e2743e49859b4750a4c50345b7fdb663]) successfully deleted
W20260810 18:14:47.681080 17545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:44543 (error 108)
I20260810 18:14:47.681715 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 17870
I20260810 18:14:47.685822 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:14:47.681874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:44543 (local address 127.0.0.1:41470)
0810 18:14:47.685613 (+ 3739us) negotiation.cc:107] Waiting for socket to connect
0810 18:14:47.685713 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:44543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3630}
W20260810 18:14:47.690909 17545 catalog_manager.cc:4584] TS dc7a1a98b3684862af61b46ffcfb27be (127.23.65.65:44543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c13ad9a9d67b42ebaca5b7a6120614da: Network error: Client connection negotiation failed: client connection to 127.23.65.65:44543: connect: Connection refused (error 111)
I20260810 18:14:47.698606 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 18088
I20260810 18:14:47.710094 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 17514
I20260810 18:15:07.722410 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38855
--webserver_interface=127.23.65.126
--webserver_port=36641
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:07.834977 18333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:07.835261 18333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:07.835299 18333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:07.839339 18333 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 18:15:07.839395 18333 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 18:15:07.839411 18333 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 18:15:07.839426 18333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:07.839447 18333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:07.839481 18333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:07.839509 18333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:07.839551 18333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:07.844883 18333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=36641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:07.845912 18333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:07.846942 18333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:07.855561 18350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:07.855571 18344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:07.855638 18333 server_base.cc:1034] running on GCE node
W20260810 18:15:07.855592 18345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:07.856611 18333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:07.857715 18333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:07.858875 18333 hybrid_clock.cc:648] HybridClock initialized: now 1786385707858851 us; error 40 us; skew 500 ppm
Aug 10 18:15:07 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:07.864782 18333 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:15:07.865720 18333 webserver.cc:460] Webserver started at http://127.23.65.126:36641/ using document root <none> and password file <none>
I20260810 18:15:07.866269 18333 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:07.866333 18333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:07.870314 18333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:15:07.872624 18358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:07.873927 18333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:07.874049 18333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "6582f75204bf4effaf30f27f8948c008"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "844a81dec5f577e1751cb0585e5193f0"
server_key_iv: "f9e517d700591b68d0b84a2d3c93889e"
server_key_version: "encryptionkey@0"
I20260810 18:15:07.874606 18333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:07.888270 18333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:07.891320 18333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:07.891500 18333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:07.898614 18333 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:38855
I20260810 18:15:07.898875 18441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:38855 every 8 connection(s)
I20260810 18:15:07.899712 18333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:07.900045 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 18333
I20260810 18:15:07.900512 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:44543
--local_ip_for_outbound_sockets=127.23.65.65
--tserver_master_addrs=127.23.65.126:38855
--webserver_port=42039
--webserver_interface=127.23.65.65
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:15:07.904163 18442 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:15:07.905787 18442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Bootstrap starting.
I20260810 18:15:07.923795 18442 log.cc:824] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:07.935072 18442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:15:07.935469 18442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: Bootstrap complete.
I20260810 18:15:07.939486 18442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:15:07.939929 18442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6582f75204bf4effaf30f27f8948c008, State: Initialized, Role: FOLLOWER
I20260810 18:15:07.940389 18442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:15:07.940513 18442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:07.940579 18442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:07.940680 18442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:15:07.942502 18442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:15:07.942816 18442 leader_election.cc:302] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6582f75204bf4effaf30f27f8948c008; no voters:
I20260810 18:15:07.943101 18442 leader_election.cc:288] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 18:15:07.943251 18446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:15:07.943495 18446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [term 3 LEADER]: Becoming Leader. State: Replica: 6582f75204bf4effaf30f27f8948c008, State: Running, Role: LEADER
I20260810 18:15:07.943853 18446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } }
I20260810 18:15:07.944273 18442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:07.945334 18449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6582f75204bf4effaf30f27f8948c008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } } }
I20260810 18:15:07.945457 18449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:07.945734 18450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6582f75204bf4effaf30f27f8948c008. Latest consensus state: current_term: 3 leader_uuid: "6582f75204bf4effaf30f27f8948c008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582f75204bf4effaf30f27f8948c008" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 38855 } } }
I20260810 18:15:07.945811 18450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:07.949199 18473 catalog_manager.cc:1259] Loaded cluster ID: bfa272cfd09047c3bf7244ce6bf547a5
I20260810 18:15:07.949259 18473 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: loading cluster ID for follower catalog manager: success
I20260810 18:15:07.951817 18473 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6582f75204bf4effaf30f27f8948c008: acquiring CA information for follower catalog manager: success
F20260810 18:15:07.953012 18473 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc9a37409ad google::LogMessage::Fail() at ??:0
@ 0x7fc9a374388a google::LogMessage::SendToLog() at ??:0
@ 0x7fc9a37404de google::LogMessage::Flush() at ??:0
@ 0x7fc9a3741a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc9a640418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc9ae0a9989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fc9ae086023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fc9ae08654b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fc9ae07f652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fc9ae07e9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fc9ae0c5541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc9a5983512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc9a6fc76db start_thread at ??:0
@ 0x7fc9a223e61f clone at ??:0
W20260810 18:15:08.036074 18444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:08.036341 18444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:08.036386 18444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:08.040541 18444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:08.040601 18444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:08.040665 18444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:15:08.045962 18444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:44543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=42039
--tserver_master_addrs=127.23.65.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:08.047048 18444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:08.048148 18444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:08.057574 18503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:08.057752 18500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:08.057574 18501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:08.058360 18444 server_base.cc:1034] running on GCE node
I20260810 18:15:08.058753 18444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:08.059382 18444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:08.060559 18444 hybrid_clock.cc:648] HybridClock initialized: now 1786385708060533 us; error 42 us; skew 500 ppm
Aug 10 18:15:08 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385708, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:08.063477 18444 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:15:08.064430 18444 webserver.cc:460] Webserver started at http://127.23.65.65:42039/ using document root <none> and password file <none>
I20260810 18:15:08.064970 18444 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:08.065038 18444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:08.068864 18444 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:15:08.070963 18514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:08.072183 18444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:08.072288 18444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "dc7a1a98b3684862af61b46ffcfb27be"
format_stamp: "Formatted at 2026-08-10 18:14:25 on dist-test-slave-jpj1"
server_key: "05ddf81bd24f310428fc3a522469e8ae"
server_key_iv: "b79c07ce88e837db9f774996a168912c"
server_key_version: "encryptionkey@0"
I20260810 18:15:08.072821 18444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:08.090349 18444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:08.093706 18444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:08.093881 18444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:08.094487 18444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:08.095649 18523 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:15:08.101145 18444 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:15:08.101226 18444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 18:15:08.101300 18444 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:15:08.103027 18444 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:15:08.103076 18444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 18:15:08.103436 18523 tablet_bootstrap.cc:492] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Bootstrap starting.
I20260810 18:15:08.113202 18444 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:44543
I20260810 18:15:08.113473 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:44543 every 8 connection(s)
I20260810 18:15:08.114303 18444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:15:08.118132 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 18444
I20260810 18:15:08.118567 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:45489
--local_ip_for_outbound_sockets=127.23.65.66
--tserver_master_addrs=127.23.65.126:38855
--webserver_port=36445
--webserver_interface=127.23.65.66
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:15:08.127830 18523 log.cc:824] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:08.134963 18523 tablet_bootstrap.cc:492] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:15:08.135342 18523 tablet_bootstrap.cc:492] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Bootstrap complete.
I20260810 18:15:08.136456 18523 ts_tablet_manager.cc:1397] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.010s
I20260810 18:15:08.138991 18523 raft_consensus.cc:348] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:15:08.139438 18523 raft_consensus.cc:729] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7a1a98b3684862af61b46ffcfb27be, State: Initialized, Role: FOLLOWER
I20260810 18:15:08.139938 18523 consensus_queue.cc:260] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:15:08.140913 18523 ts_tablet_manager.cc:1428] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260810 18:15:08.173470 18688 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:15:08.116264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:45295)
0810 18:15:08.116756 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:08.116811 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:08.117652 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:08.172454 (+ 54802us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
W20260810 18:15:08.174080 18686 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:38855 (0 consecutive failures): Network error: Failed to ping master at 127.23.65.126:38855: Client connection negotiation failed: client connection to 127.23.65.126:38855: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 18:15:08.174899 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:08.174574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:37351)
0810 18:15:08.174731 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:08.174817 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 18:15:08.175611 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:08.175366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:49037)
0810 18:15:08.175470 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:08.175522 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 18:15:08.175800 18686 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:15:08.254123 18689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:08.254374 18689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:08.254416 18689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:08.258379 18689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:08.258442 18689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:08.258500 18689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:15:08.263764 18689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:45489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=36445
--tserver_master_addrs=127.23.65.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:08.264830 18689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:08.265820 18689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:08.275146 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:08.275146 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:08.275187 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:08.275595 18689 server_base.cc:1034] running on GCE node
I20260810 18:15:08.276064 18689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:08.276685 18689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:08.277845 18689 hybrid_clock.cc:648] HybridClock initialized: now 1786385708277826 us; error 29 us; skew 500 ppm
Aug 10 18:15:08 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385708, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:08.280704 18689 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:15:08.281603 18689 webserver.cc:460] Webserver started at http://127.23.65.66:36445/ using document root <none> and password file <none>
I20260810 18:15:08.282131 18689 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:08.282197 18689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:08.286060 18689 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:08.288089 18713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:08.289340 18689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:08.289446 18689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "6c413b3c61ae44cbac0b257c54cacbb0"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "46c73ba6f351beb7482b444e218ad0a3"
server_key_iv: "34bb6cb029e2a838e176f17b759924ee"
server_key_version: "encryptionkey@0"
I20260810 18:15:08.289990 18689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:08.316638 18689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:08.319792 18689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:08.319972 18689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:08.320595 18689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:08.321460 18689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:08.321509 18689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:08.321563 18689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:08.321597 18689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:08.330133 18689 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:45489
I20260810 18:15:08.330176 18887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:45489 every 8 connection(s)
I20260810 18:15:08.331179 18689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:15:08.334688 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:08.333209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:51541)
0810 18:15:08.333878 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:08.333987 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":123,"threads_started":1}
W20260810 18:15:08.335129 18889 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:38855 (0 consecutive failures): Network error: Failed to ping master at 127.23.65.126:38855: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
I20260810 18:15:08.335755 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:08.335510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:34843)
0810 18:15:08.335631 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:08.335679 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 18:15:08.336275 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:08.336099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:44299)
0810 18:15:08.336184 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:08.336227 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 18:15:08.336426 18889 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:15:08.339221 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 18689
I20260810 18:15:08.339658 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:36889
--local_ip_for_outbound_sockets=127.23.65.67
--tserver_master_addrs=127.23.65.126:38855
--webserver_port=38983
--webserver_interface=127.23.65.67
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:08.452000 18894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:08.452289 18894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:08.452332 18894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:08.456316 18894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:08.456368 18894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:08.456423 18894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:15:08.461568 18894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:36889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=38983
--tserver_master_addrs=127.23.65.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:08.462574 18894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:08.463581 18894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:08.473616 18907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:08.473686 18902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:08.473664 18904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:08.473896 18894 server_base.cc:1034] running on GCE node
I20260810 18:15:08.474341 18894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:08.474951 18894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:08.476106 18894 hybrid_clock.cc:648] HybridClock initialized: now 1786385708476082 us; error 34 us; skew 500 ppm
Aug 10 18:15:08 dist-test-slave-jpj1 krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385708, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:08.478765 18894 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:15:08.479645 18894 webserver.cc:460] Webserver started at http://127.23.65.67:38983/ using document root <none> and password file <none>
I20260810 18:15:08.480225 18894 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:08.480290 18894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:08.484043 18894 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 18:15:08.486066 18916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:08.487282 18894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:08.487377 18894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "634a73f479994917ad451dd54ce7cddf"
format_stamp: "Formatted at 2026-08-10 18:14:26 on dist-test-slave-jpj1"
server_key: "2fe0dd5a34b3296393ec19162824b43d"
server_key_iv: "a0104bcb812bd3f089f6ac3e47a3aaa7"
server_key_version: "encryptionkey@0"
I20260810 18:15:08.487946 18894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:08.514194 18894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:08.517563 18894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:08.517728 18894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:08.518303 18894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:08.519130 18894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:08.519179 18894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:08.519227 18894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:08.519248 18894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:08.527750 18894 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:36889
I20260810 18:15:08.527879 19099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:36889 every 8 connection(s)
I20260810 18:15:08.528906 18894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:15:08.532501 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:08.530988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:46043)
0810 18:15:08.531681 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:08.531801 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":130,"threads_started":1}
W20260810 18:15:08.532992 19100 heartbeater.cc:644] Failed to heartbeat to 127.23.65.126:38855 (0 consecutive failures): Network error: Failed to ping master at 127.23.65.126:38855: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
I20260810 18:15:08.533545 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:08.533339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:43103)
0810 18:15:08.533444 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:08.533493 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 18:15:08.534086 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:08.533911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:56579)
0810 18:15:08.534000 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:08.534039 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 18:15:08.534266 19100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:15:08.536685 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 18894
I20260810 18:15:09.176877 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:09.176363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:38515)
0810 18:15:09.176640 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:09.176740 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260810 18:15:09.337462 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:09.337035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:60407)
0810 18:15:09.337290 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:09.337359 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260810 18:15:09.535303 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:09.534845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:49045)
0810 18:15:09.535136 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:09.535201 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":68,"threads_started":1}
I20260810 18:15:09.950974 19107 raft_consensus.cc:482] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:09.951177 19107 raft_consensus.cc:504] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:15:09.952332 19107 leader_election.cc:288] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489), 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889)
Aug 10 18:15:09 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385708, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.67@KRBTEST.COM
Aug 10 18:15:09 dist-test-slave-jpj1 krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385708, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.66@KRBTEST.COM
I20260810 18:15:09.966172 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:09.952605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:45489 (local address 127.23.65.65:49175)
0810 18:15:09.952935 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:09.952950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:09.953086 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:09.954595 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:09.954600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:09.955150 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:09.955659 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:09.955674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:09.956908 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:09.956910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:09.957899 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:09.957909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:09.958580 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:09.958583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:09.958782 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:09.959459 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:09.959475 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:09.961376 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:09.963415 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:09.963418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:09.963420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:09.963724 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:09.964107 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:09.964110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:09.964112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:09.964293 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:09.965036 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:09.965042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:09.965410 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 18:15:09.965631 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 18:15:09.965945 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":206,"spinlock_wait_cycles":14976,"thread_start_us":84,"threads_started":1}
I20260810 18:15:09.966254 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:09.952804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:55595 (local address 127.23.65.67:36889)
0810 18:15:09.953226 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:09.953246 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:09.953279 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:15:09.953907 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:09.953913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:09.954238 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:09.954508 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:09.955830 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:09.956771 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:09.958018 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:09.958467 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:09.958612 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:09.961257 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:09.961262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:09.961274 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:09.961301 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:09.963147 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:09.963811 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:09.963817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:09.963823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:09.963989 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:09.964390 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:09.964393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:09.964395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:09.964913 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:09.965044 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:09.965776 (+ 732us) server_negotiation.cc:299] Negotiation successful
0810 18:15:09.966061 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":84,"threads_started":1}
I20260810 18:15:09.966265 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:09.952726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:49175 (local address 127.23.65.66:45489)
0810 18:15:09.953188 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:09.953204 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:09.953244 (+ 40us) server_negotiation.cc:407] Connection header received
0810 18:15:09.953898 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:09.953904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:09.954234 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:09.954509 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:09.955830 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:09.956787 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:09.958021 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:09.958460 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:09.958612 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:09.961569 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:09.961576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:09.961592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:09.961625 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:09.963318 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:09.963811 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:09.963817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:09.963823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:09.963989 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:09.964390 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:09.964393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:09.964395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:09.964913 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:09.965044 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:09.965776 (+ 732us) server_negotiation.cc:299] Negotiation successful
0810 18:15:09.966061 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":112,"threads_started":1}
I20260810 18:15:09.966172 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:09.952611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:36889 (local address 127.23.65.65:55595)
0810 18:15:09.952988 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:09.953003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:09.953112 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:09.954593 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:09.954599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:09.955150 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:09.955660 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:09.955674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:09.956907 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:09.956910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:09.957898 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:09.957908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:09.958582 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:09.958586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:09.958784 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:09.959504 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:09.959516 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:09.961034 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:09.963262 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:09.963267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:09.963278 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:09.963707 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:09.964081 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:09.964084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:09.964087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:09.964293 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:09.965036 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:09.965042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:09.965410 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 18:15:09.965631 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 18:15:09.966010 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":356,"spinlock_wait_cycles":18048,"thread_start_us":75,"threads_started":1}
W20260810 18:15:09.968727 18516 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
W20260810 18:15:09.968881 18519 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
I20260810 18:15:09.968955 18519 leader_election.cc:302] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc7a1a98b3684862af61b46ffcfb27be; no voters: 634a73f479994917ad451dd54ce7cddf, 6c413b3c61ae44cbac0b257c54cacbb0
I20260810 18:15:09.969199 19107 raft_consensus.cc:2726] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:15:10.178205 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:10.177861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:54183)
0810 18:15:10.178016 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:10.178100 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 18:15:10.338680 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:10.338230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:48927)
0810 18:15:10.338493 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:10.338556 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260810 18:15:10.536463 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:10.536037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:52195)
0810 18:15:10.536304 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:10.536367 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
I20260810 18:15:11.179394 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:11.178961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:38551)
0810 18:15:11.179221 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:11.179289 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260810 18:15:11.339923 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:11.339456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:47077)
0810 18:15:11.339751 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:11.339821 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":106,"threads_started":1}
I20260810 18:15:11.537667 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:11.537218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:41457)
0810 18:15:11.537484 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:11.537543 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260810 18:15:11.980623 19123 raft_consensus.cc:482] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:11.980769 19123 raft_consensus.cc:504] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:15:11.981146 19123 leader_election.cc:288] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489), 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889)
W20260810 18:15:11.981983 18516 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
W20260810 18:15:11.982102 18519 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
I20260810 18:15:11.982178 18519 leader_election.cc:302] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc7a1a98b3684862af61b46ffcfb27be; no voters: 634a73f479994917ad451dd54ce7cddf, 6c413b3c61ae44cbac0b257c54cacbb0
I20260810 18:15:11.982314 19123 raft_consensus.cc:2726] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:15:12.180644 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:12.180197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:33543)
0810 18:15:12.180470 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:12.180535 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260810 18:15:12.341024 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:12.340603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:49923)
0810 18:15:12.340863 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:12.340920 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260810 18:15:12.538832 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:12.538393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:59765)
0810 18:15:12.538655 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:12.538735 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260810 18:15:13.181998 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:13.181548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:45579)
0810 18:15:13.181842 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:13.181898 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":82,"threads_started":1}
I20260810 18:15:13.342172 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:13.341760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:39819)
0810 18:15:13.342017 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:13.342074 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260810 18:15:13.539965 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:13.539578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:44853)
0810 18:15:13.539817 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:13.539870 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260810 18:15:13.839828 19135 raft_consensus.cc:482] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:13.839970 19135 raft_consensus.cc:504] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:15:13.840418 19135 leader_election.cc:288] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489), 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889)
W20260810 18:15:13.841259 18516 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
W20260810 18:15:13.841382 18519 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
I20260810 18:15:13.841478 18519 leader_election.cc:302] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc7a1a98b3684862af61b46ffcfb27be; no voters: 634a73f479994917ad451dd54ce7cddf, 6c413b3c61ae44cbac0b257c54cacbb0
I20260810 18:15:13.841622 19135 raft_consensus.cc:2726] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:15:14.183194 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:14.182726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:33365)
0810 18:15:14.183012 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:14.183089 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260810 18:15:14.343242 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:14.342845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:47395)
0810 18:15:14.343089 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:14.343143 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260810 18:15:14.541082 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:14.540663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:41483)
0810 18:15:14.540933 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:14.540988 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260810 18:15:15.184391 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:15.183957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:54363)
0810 18:15:15.184230 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:15.184287 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260810 18:15:15.344403 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:15.343962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:52541)
0810 18:15:15.344254 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:15.344307 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":70,"threads_started":1}
I20260810 18:15:15.542353 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:15.541855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:55925)
0810 18:15:15.542197 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:15.542254 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":95,"threads_started":1}
I20260810 18:15:16.185567 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:16.185099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:33815)
0810 18:15:16.185401 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:16.185462 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":114,"threads_started":1}
I20260810 18:15:16.345520 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:16.345080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:59593)
0810 18:15:16.345363 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:16.345421 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
I20260810 18:15:16.543761 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:16.543246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:48877)
0810 18:15:16.543566 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:16.543636 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":95,"threads_started":1}
I20260810 18:15:17.186825 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:17.186334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:57859)
0810 18:15:17.186652 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:17.186717 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
I20260810 18:15:17.346715 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:17.346296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:59047)
0810 18:15:17.346547 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:17.346608 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260810 18:15:17.544978 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:17.544513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:49537)
0810 18:15:17.544804 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:17.544876 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260810 18:15:18.188326 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:18.187771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:58825)
0810 18:15:18.188122 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:18.188197 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
I20260810 18:15:18.347960 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:18.347498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:43945)
0810 18:15:18.347797 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:18.347857 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":116,"threads_started":1}
I20260810 18:15:18.546187 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:18.545759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:41415)
0810 18:15:18.546037 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:18.546090 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260810 18:15:18.631744 19159 raft_consensus.cc:482] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:18.631910 19159 raft_consensus.cc:504] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:15:18.632364 19159 leader_election.cc:288] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489), 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889)
W20260810 18:15:18.633062 18516 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
W20260810 18:15:18.633199 18519 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
I20260810 18:15:18.633255 18519 leader_election.cc:302] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc7a1a98b3684862af61b46ffcfb27be; no voters: 634a73f479994917ad451dd54ce7cddf, 6c413b3c61ae44cbac0b257c54cacbb0
I20260810 18:15:18.633409 19159 raft_consensus.cc:2726] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:15:19.189487 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:19.189054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:50235)
0810 18:15:19.189309 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:19.189371 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260810 18:15:19.349045 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:19.348656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:41329)
0810 18:15:19.348893 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:19.348946 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260810 18:15:19.547403 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:19.546984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:40527)
0810 18:15:19.547247 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:19.547303 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":94,"threads_started":1}
I20260810 18:15:20.190726 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:20.190284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:57351)
0810 18:15:20.190535 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:20.190600 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260810 18:15:20.350226 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:20.349775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:55067)
0810 18:15:20.350056 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:20.350115 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260810 18:15:20.548688 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:20.548205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:43439)
0810 18:15:20.548490 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:20.548555 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260810 18:15:21.192147 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:21.191672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:36371)
0810 18:15:21.191961 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:21.192031 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260810 18:15:21.351442 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:21.350977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:42989)
0810 18:15:21.351275 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:21.351336 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
I20260810 18:15:21.549935 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:21.549489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:40529)
0810 18:15:21.549766 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:21.549823 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20260810 18:15:22.011922 19175 raft_consensus.cc:482] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:22.012068 19175 raft_consensus.cc:504] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c413b3c61ae44cbac0b257c54cacbb0" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45489 } } peers { permanent_uuid: "634a73f479994917ad451dd54ce7cddf" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 36889 } } peers { permanent_uuid: "dc7a1a98b3684862af61b46ffcfb27be" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 44543 } }
I20260810 18:15:22.012494 19175 leader_election.cc:288] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489), 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889)
W20260810 18:15:22.013345 18516 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c413b3c61ae44cbac0b257c54cacbb0 (127.23.65.66:45489): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
W20260810 18:15:22.013479 18519 leader_election.cc:341] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 634a73f479994917ad451dd54ce7cddf (127.23.65.67:36889): Not found: Tablet not found: c13ad9a9d67b42ebaca5b7a6120614da
I20260810 18:15:22.013568 18519 leader_election.cc:302] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc7a1a98b3684862af61b46ffcfb27be; no voters: 634a73f479994917ad451dd54ce7cddf, 6c413b3c61ae44cbac0b257c54cacbb0
I20260810 18:15:22.013741 19175 raft_consensus.cc:2726] T c13ad9a9d67b42ebaca5b7a6120614da P dc7a1a98b3684862af61b46ffcfb27be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:15:22.193392 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:22.192873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:36631)
0810 18:15:22.193133 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:22.193285 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260810 18:15:22.352612 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:22.352192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:40699)
0810 18:15:22.352449 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:22.352510 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260810 18:15:22.551152 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:22.550678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:46863)
0810 18:15:22.550970 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:22.551034 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260810 18:15:23.194852 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:23.194345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.65:59949)
0810 18:15:23.194692 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:23.194752 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":80,"threads_started":1}
I20260810 18:15:23.353945 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:23.353516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.66:53927)
0810 18:15:23.353784 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:23.353843 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260810 18:15:23.538691 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 18444
I20260810 18:15:23.552490 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:23.552099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:38855 (local address 127.23.65.67:39441)
0810 18:15:23.552338 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:23.552393 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":83,"threads_started":1}
I20260810 18:15:23.553074 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 18689
I20260810 18:15:23.563604 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 18894
2026-08-10T18:15:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59444 ms)
[----------] 30 tests from SecurityITest (244784 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:23 dist-test-slave-jpj1 krb5kdc[19184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:23 dist-test-slave-jpj1 krb5kdc[19184](info): set up 2 sockets
Aug 10 18:15:23 dist-test-slave-jpj1 krb5kdc[19184](info): commencing operation
krb5kdc: starting...
W20260810 18:15:24.726526 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:24 dist-test-slave-jpj1 krb5kdc[19184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:15:24.848855 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:36951
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:36951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:24.957702 19202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:24.957989 19202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:24.958045 19202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:24.962054 19202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:24.962117 19202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:24.962148 19202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:24.962168 19202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:24.962189 19202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:24.967465 19202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:36951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:36951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:24.968531 19202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:24.969560 19202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:24.978034 19212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:24.978142 19202 server_base.cc:1034] running on GCE node
W20260810 18:15:24.978070 19210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:24.978072 19214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:24.979100 19202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:24.980211 19202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:24.981380 19202 hybrid_clock.cc:648] HybridClock initialized: now 1786385724981364 us; error 27 us; skew 500 ppm
Aug 10 18:15:24 dist-test-slave-jpj1 krb5kdc[19184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385724, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:24.984079 19202 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:15:24.984947 19202 webserver.cc:460] Webserver started at http://127.23.65.126:36009/ using document root <none> and password file <none>
I20260810 18:15:24.985481 19202 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:24.985548 19202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:24.985749 19202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:24.987787 19202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "0b1aa70006b04d49b0c893fb705a3246"
format_stamp: "Formatted at 2026-08-10 18:15:24 on dist-test-slave-jpj1"
server_key: "92b9aab793c57a02b2e2409db4fa1523"
server_key_iv: "ef06d23e8dea2bd227234628d5168c4c"
server_key_version: "encryptionkey@0"
I20260810 18:15:24.988339 19202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "0b1aa70006b04d49b0c893fb705a3246"
format_stamp: "Formatted at 2026-08-10 18:15:24 on dist-test-slave-jpj1"
server_key: "92b9aab793c57a02b2e2409db4fa1523"
server_key_iv: "ef06d23e8dea2bd227234628d5168c4c"
server_key_version: "encryptionkey@0"
I20260810 18:15:24.991925 19202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:15:24.994230 19223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:24.995362 19202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:24.995473 19202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "0b1aa70006b04d49b0c893fb705a3246"
format_stamp: "Formatted at 2026-08-10 18:15:24 on dist-test-slave-jpj1"
server_key: "92b9aab793c57a02b2e2409db4fa1523"
server_key_iv: "ef06d23e8dea2bd227234628d5168c4c"
server_key_version: "encryptionkey@0"
I20260810 18:15:24.995618 19202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:25.009110 19202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:25.012220 19202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:25.012403 19202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:25.019199 19202 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:36951
I20260810 18:15:25.019250 19301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:36951 every 8 connection(s)
I20260810 18:15:25.020175 19202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:25.022948 19303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:25.024724 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 19202
I20260810 18:15:25.024876 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:25.025151 23813 external_mini_cluster.cc:1428] Setting key b893809db9ef502898c86ab79ed03f09
I20260810 18:15:25.029811 19303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246: Bootstrap starting.
I20260810 18:15:25.032492 19303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:25.033267 19303 log.cc:824] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:15:25 dist-test-slave-jpj1 krb5kdc[19184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:25.035738 19303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246: No bootstrap required, opened a new log
I20260810 18:15:25.040412 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.026533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.23.65.126:36951)
0810 18:15:25.026985 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.027010 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.027057 (+ 47us) server_negotiation.cc:407] Connection header received
0810 18:15:25.027872 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.027898 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.028325 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.028707 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:25.029331 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.030773 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.031671 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.032354 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.032522 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.035355 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:25.035361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:25.035379 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:25.035410 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:25.037321 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.037754 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.037760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.037766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.037916 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.038152 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.038156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.038157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.038694 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:25.038889 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.039084 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.039387 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
I20260810 18:15:25.040374 19303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1aa70006b04d49b0c893fb705a3246" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36951 } }
I20260810 18:15:25.040647 19303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:25.040707 19303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1aa70006b04d49b0c893fb705a3246, State: Initialized, Role: FOLLOWER
I20260810 18:15:25.041203 19303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1aa70006b04d49b0c893fb705a3246" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36951 } }
I20260810 18:15:25.041376 19303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:25.041443 19303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:25.041543 19303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:25.042436 19303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1aa70006b04d49b0c893fb705a3246" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36951 } }
I20260810 18:15:25.042804 19303 leader_election.cc:302] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b1aa70006b04d49b0c893fb705a3246; no voters:
I20260810 18:15:25.043097 19303 leader_election.cc:288] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:25.043201 19308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:25.043411 19308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [term 1 LEADER]: Becoming Leader. State: Replica: 0b1aa70006b04d49b0c893fb705a3246, State: Running, Role: LEADER
I20260810 18:15:25.043773 19308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1aa70006b04d49b0c893fb705a3246" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36951 } }
I20260810 18:15:25.044229 19303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:25.045222 19309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b1aa70006b04d49b0c893fb705a3246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1aa70006b04d49b0c893fb705a3246" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36951 } } }
I20260810 18:15:25.045439 19309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:25.045535 19311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b1aa70006b04d49b0c893fb705a3246. Latest consensus state: current_term: 1 leader_uuid: "0b1aa70006b04d49b0c893fb705a3246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1aa70006b04d49b0c893fb705a3246" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36951 } } }
I20260810 18:15:25.045611 19311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246 [sys.catalog]: This master's current role is: LEADER
W20260810 18:15:25.049000 19335 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:15:25.049072 19335 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:15:25.049126 19325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:25.049990 19325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:25.054816 19325 catalog_manager.cc:1347] Generated new cluster ID: 4b2b26e1466540dabe6c9a24179875ed
I20260810 18:15:25.054881 19325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:25.079974 19325 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:25.081399 19325 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:25.087868 19325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246: Generated new TSK 0
I20260810 18:15:25.088605 19325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:15:25.126137 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:36951
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:25.238585 19346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:25.238858 19346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:25.238903 19346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:25.242933 19346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:25.242993 19346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:25.243049 19346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:15:25.248418 19346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:25.249429 19346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:25.250439 19346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:25.259805 19358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:25.259816 19361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:25.259991 19346 server_base.cc:1034] running on GCE node
W20260810 18:15:25.259912 19364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:25.260507 19346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:25.261159 19346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:25.262334 19346 hybrid_clock.cc:648] HybridClock initialized: now 1786385725262318 us; error 31 us; skew 500 ppm
Aug 10 18:15:25 dist-test-slave-jpj1 krb5kdc[19184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:25.265358 19346 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:15:25.266341 19346 webserver.cc:460] Webserver started at http://127.23.65.65:41221/ using document root <none> and password file <none>
I20260810 18:15:25.266952 19346 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:25.267021 19346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:25.267215 19346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:25.269258 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "25901419426349efae35591febf1354c"
format_stamp: "Formatted at 2026-08-10 18:15:25 on dist-test-slave-jpj1"
server_key: "8c92aa29b60e83ff90cf8fee5cf2b267"
server_key_iv: "e7e465ba78a8d021e7f99a0cc052f65d"
server_key_version: "encryptionkey@0"
I20260810 18:15:25.269815 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "25901419426349efae35591febf1354c"
format_stamp: "Formatted at 2026-08-10 18:15:25 on dist-test-slave-jpj1"
server_key: "8c92aa29b60e83ff90cf8fee5cf2b267"
server_key_iv: "e7e465ba78a8d021e7f99a0cc052f65d"
server_key_version: "encryptionkey@0"
I20260810 18:15:25.273506 19346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:15:25.275830 19376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:25.276975 19346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:15:25.277084 19346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "25901419426349efae35591febf1354c"
format_stamp: "Formatted at 2026-08-10 18:15:25 on dist-test-slave-jpj1"
server_key: "8c92aa29b60e83ff90cf8fee5cf2b267"
server_key_iv: "e7e465ba78a8d021e7f99a0cc052f65d"
server_key_version: "encryptionkey@0"
I20260810 18:15:25.277179 19346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:25.300561 19346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:25.303417 19346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:25.303617 19346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:25.304210 19346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:25.305043 19346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:25.305092 19346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:25.305146 19346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:25.305176 19346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:25.313706 19346 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:37517
I20260810 18:15:25.313736 19561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:37517 every 8 connection(s)
I20260810 18:15:25.314550 19346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
Aug 10 18:15:25 dist-test-slave-jpj1 krb5kdc[19184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:25.323278 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 19346
I20260810 18:15:25.323474 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:15:25.323791 23813 external_mini_cluster.cc:1428] Setting key a6b880039c24a9d5bae5a5c476d8984d
I20260810 18:15:25.327461 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.316267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:48195 (local address 127.23.65.126:36951)
0810 18:15:25.316386 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.316390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.317365 (+ 975us) server_negotiation.cc:407] Connection header received
0810 18:15:25.318181 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.318183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.318232 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.318300 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:25.319400 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.319965 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.321004 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.321215 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.321290 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.323637 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:25.323641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:25.323643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:25.323669 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:25.325451 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.325997 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.326000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.326001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.326069 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.326386 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.326388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.326389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.326544 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:25.326612 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.327173 (+ 561us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.327300 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:15:25.328310 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.316590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:36951 (local address 127.23.65.65:48195)
0810 18:15:25.317205 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:25.317262 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:25.318004 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:25.318449 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:25.318456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:25.318797 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:25.319284 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.319298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.320052 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.320054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.320914 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.320923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.321321 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.321323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.321457 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.321960 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:25.321976 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:25.323435 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:25.325567 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:25.325573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:25.325589 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:25.325914 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:25.326138 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:25.326140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:25.326142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:25.326314 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:25.326628 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:25.326634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:25.326929 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 18:15:25.327130 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 18:15:25.327451 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":109,"threads_started":1}
I20260810 18:15:25.329449 19562 heartbeater.cc:347] Connected to a master server at 127.23.65.126:36951
I20260810 18:15:25.329684 19562 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:25.330176 19562 heartbeater.cc:511] Master 127.23.65.126:36951 requested a full tablet report, sending...
I20260810 18:15:25.331642 19252 ts_manager.cc:194] Registered new tserver with Master: 25901419426349efae35591febf1354c (127.23.65.65:37517)
I20260810 18:15:25.332729 19252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:48195
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:15:25.350685 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:36951
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:25.462862 19568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:25.463127 19568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:25.463188 19568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:25.467168 19568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:25.467227 19568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:25.467306 19568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:15:25.472569 19568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:25.473590 19568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:25.474583 19568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:25.483955 19579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:25.483994 19582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:25.483983 19580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:25.484264 19568 server_base.cc:1034] running on GCE node
I20260810 18:15:25.484699 19568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:25.485317 19568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:25.486462 19568 hybrid_clock.cc:648] HybridClock initialized: now 1786385725486440 us; error 35 us; skew 500 ppm
Aug 10 18:15:25 dist-test-slave-jpj1 krb5kdc[19184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:25.489168 19568 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:15:25.490083 19568 webserver.cc:460] Webserver started at http://127.23.65.66:41879/ using document root <none> and password file <none>
I20260810 18:15:25.490661 19568 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:25.490732 19568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:25.490921 19568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:25.493048 19568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "9d5574e389d84b8787efee51cc355cff"
format_stamp: "Formatted at 2026-08-10 18:15:25 on dist-test-slave-jpj1"
server_key: "b4cc65eeabacd315df13bdf1869277ee"
server_key_iv: "e16ff84c7f9c2a37c53acad5085969c0"
server_key_version: "encryptionkey@0"
I20260810 18:15:25.493603 19568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "9d5574e389d84b8787efee51cc355cff"
format_stamp: "Formatted at 2026-08-10 18:15:25 on dist-test-slave-jpj1"
server_key: "b4cc65eeabacd315df13bdf1869277ee"
server_key_iv: "e16ff84c7f9c2a37c53acad5085969c0"
server_key_version: "encryptionkey@0"
I20260810 18:15:25.497418 19568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:15:25.499930 19592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:25.501104 19568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:25.501215 19568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "9d5574e389d84b8787efee51cc355cff"
format_stamp: "Formatted at 2026-08-10 18:15:25 on dist-test-slave-jpj1"
server_key: "b4cc65eeabacd315df13bdf1869277ee"
server_key_iv: "e16ff84c7f9c2a37c53acad5085969c0"
server_key_version: "encryptionkey@0"
I20260810 18:15:25.501320 19568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:25.522588 19568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:25.525519 19568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:25.525699 19568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:25.526356 19568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:25.527213 19568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:25.527271 19568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:25.527329 19568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:25.527355 19568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:25.536048 19568 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:34121
I20260810 18:15:25.536079 19765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:34121 every 8 connection(s)
I20260810 18:15:25.536940 19568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
Aug 10 18:15:25 dist-test-slave-jpj1 krb5kdc[19184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:25.546492 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 19568
I20260810 18:15:25.546622 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:15:25.546903 23813 external_mini_cluster.cc:1428] Setting key 9ee64fc48186f93ff53997dbacb85dc4
I20260810 18:15:25.550063 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.538695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:36451 (local address 127.23.65.126:36951)
0810 18:15:25.538831 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.538835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.539966 (+ 1131us) server_negotiation.cc:407] Connection header received
0810 18:15:25.540853 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.540856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.540905 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.540971 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:25.542095 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.542616 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.543815 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.544059 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.544140 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.546620 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:25.546624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:25.546627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:25.546654 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:25.548083 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.548614 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.548618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.548620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.548730 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.549019 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.549022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.549023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.549245 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:25.549341 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.549794 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.549911 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:15:25.550863 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.539080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:36951 (local address 127.23.65.66:36451)
0810 18:15:25.539812 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:25.539865 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:25.540648 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:25.541122 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:25.541128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:25.541484 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:25.541974 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.541988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.542707 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.542709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.543628 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.543640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.544185 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.544187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.544335 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.544930 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:25.544950 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:25.546462 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:25.548210 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:25.548215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:25.548231 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:25.548512 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:25.548815 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:25.548817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:25.548818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:25.548946 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:25.549329 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:25.549334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:25.549563 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 18:15:25.549726 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 18:15:25.549979 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":120,"threads_started":1}
I20260810 18:15:25.552012 19766 heartbeater.cc:347] Connected to a master server at 127.23.65.126:36951
I20260810 18:15:25.552250 19766 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:25.552717 19766 heartbeater.cc:511] Master 127.23.65.126:36951 requested a full tablet report, sending...
I20260810 18:15:25.553741 19252 ts_manager.cc:194] Registered new tserver with Master: 9d5574e389d84b8787efee51cc355cff (127.23.65.66:34121)
I20260810 18:15:25.554291 19252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:36451
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:15:25.574752 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:36951
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:25.686380 19772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:25.686640 19772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:25.686699 19772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:25.690668 19772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:25.690724 19772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:25.690802 19772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:15:25.695889 19772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:25.696909 19772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:25.697896 19772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:25.707212 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:25.707247 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:25.707367 19772 server_base.cc:1034] running on GCE node
W20260810 18:15:25.707207 19786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:25.707944 19772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:25.708521 19772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:25.709677 19772 hybrid_clock.cc:648] HybridClock initialized: now 1786385725709650 us; error 39 us; skew 500 ppm
Aug 10 18:15:25 dist-test-slave-jpj1 krb5kdc[19184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:25.712307 19772 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:15:25.713115 19772 webserver.cc:460] Webserver started at http://127.23.65.67:37587/ using document root <none> and password file <none>
I20260810 18:15:25.713604 19772 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:25.713660 19772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:25.713830 19772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:25.715811 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "dd611055d5a64e098604e5afa07af491"
format_stamp: "Formatted at 2026-08-10 18:15:25 on dist-test-slave-jpj1"
server_key: "17fbc43a39f184667ec575d14c81d7ee"
server_key_iv: "5ab9e0635b3ee10b8e2b81b0ed92e62f"
server_key_version: "encryptionkey@0"
I20260810 18:15:25.716331 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "dd611055d5a64e098604e5afa07af491"
format_stamp: "Formatted at 2026-08-10 18:15:25 on dist-test-slave-jpj1"
server_key: "17fbc43a39f184667ec575d14c81d7ee"
server_key_iv: "5ab9e0635b3ee10b8e2b81b0ed92e62f"
server_key_version: "encryptionkey@0"
I20260810 18:15:25.719870 19772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:15:25.722159 19798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:25.723310 19772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:25.723417 19772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "dd611055d5a64e098604e5afa07af491"
format_stamp: "Formatted at 2026-08-10 18:15:25 on dist-test-slave-jpj1"
server_key: "17fbc43a39f184667ec575d14c81d7ee"
server_key_iv: "5ab9e0635b3ee10b8e2b81b0ed92e62f"
server_key_version: "encryptionkey@0"
I20260810 18:15:25.723516 19772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:25.739027 19772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:25.741719 19772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:25.741891 19772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:25.742451 19772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:25.743237 19772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:25.743284 19772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:25.743327 19772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:25.743356 19772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:25.751621 19772 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:43283
I20260810 18:15:25.751683 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:43283 every 8 connection(s)
I20260810 18:15:25.752444 19772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:15:25.761014 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 19772
I20260810 18:15:25.761164 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
Aug 10 18:15:25 dist-test-slave-jpj1 krb5kdc[19184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:25.761494 23813 external_mini_cluster.cc:1428] Setting key 3dd1ee1013dbae4c54ef5ffb66abfdc4
I20260810 18:15:25.765254 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.754084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:60543 (local address 127.23.65.126:36951)
0810 18:15:25.754202 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.754206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.755178 (+ 972us) server_negotiation.cc:407] Connection header received
0810 18:15:25.756060 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.756062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.756110 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.756179 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:25.757307 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.758058 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.759161 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.759372 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.759438 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.761911 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:25.761914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:25.761916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:25.761941 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:25.763224 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.763778 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.763781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.763782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.763858 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.764188 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.764191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.764192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.764376 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:25.764460 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.764964 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.765082 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 18:15:25.766009 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.754413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:36951 (local address 127.23.65.67:60543)
0810 18:15:25.755034 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:25.755073 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:25.755884 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:25.756340 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:25.756346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:25.756676 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:25.757144 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.757161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.758184 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.758188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.759061 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.759070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.759478 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.759481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.759661 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.760178 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:25.760194 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:25.761738 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:25.763338 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:25.763344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:25.763358 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:25.763677 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:25.763941 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:25.763944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:25.763946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:25.764085 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:25.764476 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:25.764483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:25.764729 (+ 246us) client_negotiation.cc:769] Sending connection context
0810 18:15:25.764888 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 18:15:25.765207 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":113,"threads_started":1}
I20260810 18:15:25.767009 19982 heartbeater.cc:347] Connected to a master server at 127.23.65.126:36951
I20260810 18:15:25.767246 19982 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:25.767763 19982 heartbeater.cc:511] Master 127.23.65.126:36951 requested a full tablet report, sending...
I20260810 18:15:25.768721 19252 ts_manager.cc:194] Registered new tserver with Master: dd611055d5a64e098604e5afa07af491 (127.23.65.67:43283)
I20260810 18:15:25.769280 19252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:60543
I20260810 18:15:25.775007 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:15:25.784046 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.777023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.23.65.126:36951)
0810 18:15:25.777157 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.777162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.777341 (+ 179us) server_negotiation.cc:407] Connection header received
0810 18:15:25.777468 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.777471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.777514 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.777575 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:25.778248 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.778772 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.779605 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.779831 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.779914 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.781188 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:25.781192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:25.781194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:25.781221 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:25.782448 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.782832 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.782836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.782837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.782921 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:25.783166 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:25.783169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:25.783170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:25.783395 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:25.783513 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.783781 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.783893 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:15:25.786616 19252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:15:25.789614 19252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:15:25.803092 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.798477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.23.65.66:34121)
0810 18:15:25.798879 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.798889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.798906 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:15:25.798968 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.798972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.799109 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.799258 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:25.799749 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.800817 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.801803 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.802443 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.802535 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.802681 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.802845 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.802937 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":133,"threads_started":1}
I20260810 18:15:25.803265 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.797415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:43283 (local address 127.0.0.1:50102)
0810 18:15:25.797913 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:25.797926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:25.798006 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:25.799252 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:25.799256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:25.799275 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:25.799360 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.799366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.800787 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.800791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.801828 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.801837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.802866 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.802870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.802990 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.803014 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:15:25.803071 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 18:15:25.803133 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":19,"thread_start_us":119,"threads_started":1}
I20260810 18:15:25.803351 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.798218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.23.65.67:43283)
0810 18:15:25.798785 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.798797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.798821 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:15:25.798895 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.798901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.799077 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.799255 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:25.799472 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.800642 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.801931 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.802722 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.802945 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.803056 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.803105 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.803191 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":98,"threads_started":1}
I20260810 18:15:25.803475 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.797324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37517 (local address 127.0.0.1:47834)
0810 18:15:25.797763 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:25.797810 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:25.797938 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:25.798513 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:25.798519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:25.798539 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:25.798852 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.798860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.800247 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.800250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.801945 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.801953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.803115 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.803119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.803204 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.803226 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:15:25.803294 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:15:25.803363 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":65,"threads_started":1}
I20260810 18:15:25.803896 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.797441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.23.65.65:37517)
0810 18:15:25.797782 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.797800 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.798070 (+ 270us) server_negotiation.cc:407] Connection header received
0810 18:15:25.798146 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.798151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.798369 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.798523 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:25.799039 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.800123 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.802053 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.802821 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.802912 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.803054 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.803665 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.803760 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":54,"threads_started":1}
I20260810 18:15:25.804097 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.798009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:34121 (local address 127.0.0.1:37938)
0810 18:15:25.798259 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:25.798272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:25.798349 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:25.799354 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:25.799357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:25.799375 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:25.799473 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.799480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.800934 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.800938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.801683 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.801691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.802549 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.802552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.802656 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.802693 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 18:15:25.802798 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 18:15:25.803977 (+ 1179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260810 18:15:25.806480 19458 tablet_service.cc:1467] Processing CreateTablet for tablet af4a79e68ca44c23bc763b3bda008b6d (DEFAULT_TABLE table=test-table [id=9a54d231e81a4223a8cefa8faaaffe54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:25.806509 19877 tablet_service.cc:1467] Processing CreateTablet for tablet af4a79e68ca44c23bc763b3bda008b6d (DEFAULT_TABLE table=test-table [id=9a54d231e81a4223a8cefa8faaaffe54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:25.806767 19668 tablet_service.cc:1467] Processing CreateTablet for tablet af4a79e68ca44c23bc763b3bda008b6d (DEFAULT_TABLE table=test-table [id=9a54d231e81a4223a8cefa8faaaffe54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:25.807463 19458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4a79e68ca44c23bc763b3bda008b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:25.807582 19877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4a79e68ca44c23bc763b3bda008b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:25.807616 19668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4a79e68ca44c23bc763b3bda008b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:25.813796 20010 tablet_bootstrap.cc:492] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c: Bootstrap starting.
I20260810 18:15:25.814927 20013 tablet_bootstrap.cc:492] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: Bootstrap starting.
I20260810 18:15:25.814929 20014 tablet_bootstrap.cc:492] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: Bootstrap starting.
I20260810 18:15:25.816020 20010 tablet_bootstrap.cc:654] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:25.817018 20014 tablet_bootstrap.cc:654] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:25.817044 20010 log.cc:824] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:25.817468 20013 tablet_bootstrap.cc:654] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:25.818214 20014 log.cc:824] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:25.818214 20013 log.cc:824] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:25.819128 20010 tablet_bootstrap.cc:492] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c: No bootstrap required, opened a new log
I20260810 18:15:25.819317 20010 ts_tablet_manager.cc:1397] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:15:25.820127 20013 tablet_bootstrap.cc:492] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: No bootstrap required, opened a new log
I20260810 18:15:25.820127 20014 tablet_bootstrap.cc:492] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: No bootstrap required, opened a new log
I20260810 18:15:25.820305 20013 ts_tablet_manager.cc:1397] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:15:25.820305 20014 ts_tablet_manager.cc:1397] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:15:25.822154 20010 raft_consensus.cc:348] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } }
I20260810 18:15:25.822371 20010 raft_consensus.cc:374] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:25.822453 20010 raft_consensus.cc:729] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25901419426349efae35591febf1354c, State: Initialized, Role: FOLLOWER
I20260810 18:15:25.822863 20010 consensus_queue.cc:260] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } }
I20260810 18:15:25.823529 19562 heartbeater.cc:503] Master 127.23.65.126:36951 was elected leader, sending a full tablet report...
I20260810 18:15:25.823835 20010 ts_tablet_manager.cc:1428] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 18:15:25.824038 20013 raft_consensus.cc:348] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } }
I20260810 18:15:25.824313 20013 raft_consensus.cc:374] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:25.824370 20013 raft_consensus.cc:729] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5574e389d84b8787efee51cc355cff, State: Initialized, Role: FOLLOWER
I20260810 18:15:25.824779 20013 consensus_queue.cc:260] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } }
I20260810 18:15:25.825145 20014 raft_consensus.cc:348] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } }
I20260810 18:15:25.825448 20014 raft_consensus.cc:374] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:25.825543 20014 raft_consensus.cc:729] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd611055d5a64e098604e5afa07af491, State: Initialized, Role: FOLLOWER
I20260810 18:15:25.825775 19766 heartbeater.cc:503] Master 127.23.65.126:36951 was elected leader, sending a full tablet report...
I20260810 18:15:25.826025 20014 consensus_queue.cc:260] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } }
I20260810 18:15:25.826327 20013 ts_tablet_manager.cc:1428] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:15:25.826943 19982 heartbeater.cc:503] Master 127.23.65.126:36951 was elected leader, sending a full tablet report...
I20260810 18:15:25.826996 20014 ts_tablet_manager.cc:1428] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:15:25.965154 20027 raft_consensus.cc:482] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:25.965370 20027 raft_consensus.cc:504] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } }
I20260810 18:15:25.966516 20027 leader_election.cc:288] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25901419426349efae35591febf1354c (127.23.65.65:37517), dd611055d5a64e098604e5afa07af491 (127.23.65.67:43283)
I20260810 18:15:25.971179 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.967055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:42317 (local address 127.23.65.65:37517)
0810 18:15:25.967179 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.967183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.967195 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:15:25.967242 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.967257 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.967297 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.967358 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:25.967705 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.968477 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.970001 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.970592 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.970708 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.970762 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.970985 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.971052 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:15:25.971179 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.966922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:49447 (local address 127.23.65.67:43283)
0810 18:15:25.967049 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:25.967053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:25.967461 (+ 408us) server_negotiation.cc:407] Connection header received
0810 18:15:25.967590 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:25.967593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:25.967638 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:25.967710 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:25.967926 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.968621 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.969994 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.970592 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.970709 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.970766 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:25.970985 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 18:15:25.971049 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:15:25.971217 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.966772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37517 (local address 127.23.65.66:42317)
0810 18:15:25.966885 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:25.966900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:25.966992 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:25.967401 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:25.967404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:25.967427 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:25.967597 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.967604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.968577 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.968580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.969878 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.969890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.970697 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.970700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.970852 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.970890 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 18:15:25.970965 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:15:25.971069 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":73,"spinlock_wait_cycles":16000}
I20260810 18:15:25.971226 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:25.966776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:43283 (local address 127.23.65.66:49447)
0810 18:15:25.967350 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:25.967364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:25.967484 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:25.967728 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:25.967730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:25.967751 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:25.967836 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.967840 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.968708 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.968711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.969899 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:25.969906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:25.970697 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:25.970700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:25.970856 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:25.970896 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 18:15:25.970964 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:15:25.971063 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":248,"spinlock_wait_cycles":384,"thread_start_us":144,"threads_started":1}
I20260810 18:15:25.971881 19493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4a79e68ca44c23bc763b3bda008b6d" candidate_uuid: "9d5574e389d84b8787efee51cc355cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25901419426349efae35591febf1354c" is_pre_election: true
I20260810 18:15:25.971940 19901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4a79e68ca44c23bc763b3bda008b6d" candidate_uuid: "9d5574e389d84b8787efee51cc355cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd611055d5a64e098604e5afa07af491" is_pre_election: true
I20260810 18:15:25.972180 19493 raft_consensus.cc:2454] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d5574e389d84b8787efee51cc355cff in term 0.
I20260810 18:15:25.972214 19901 raft_consensus.cc:2454] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d5574e389d84b8787efee51cc355cff in term 0.
I20260810 18:15:25.972664 19597 leader_election.cc:302] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25901419426349efae35591febf1354c, 9d5574e389d84b8787efee51cc355cff; no voters:
I20260810 18:15:25.972905 20027 raft_consensus.cc:2781] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:15:25.972997 20027 raft_consensus.cc:482] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:15:25.973048 20027 raft_consensus.cc:3037] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:25.974064 20027 raft_consensus.cc:504] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } }
I20260810 18:15:25.974378 20027 leader_election.cc:288] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [CANDIDATE]: Term 1 election: Requested vote from peers 25901419426349efae35591febf1354c (127.23.65.65:37517), dd611055d5a64e098604e5afa07af491 (127.23.65.67:43283)
I20260810 18:15:25.974718 19901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4a79e68ca44c23bc763b3bda008b6d" candidate_uuid: "9d5574e389d84b8787efee51cc355cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd611055d5a64e098604e5afa07af491"
I20260810 18:15:25.974718 19493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4a79e68ca44c23bc763b3bda008b6d" candidate_uuid: "9d5574e389d84b8787efee51cc355cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25901419426349efae35591febf1354c"
I20260810 18:15:25.974848 19901 raft_consensus.cc:3037] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:25.974867 19493 raft_consensus.cc:3037] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:25.976176 19901 raft_consensus.cc:2454] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d5574e389d84b8787efee51cc355cff in term 1.
I20260810 18:15:25.976176 19493 raft_consensus.cc:2454] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d5574e389d84b8787efee51cc355cff in term 1.
I20260810 18:15:25.976492 19597 leader_election.cc:302] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25901419426349efae35591febf1354c, 9d5574e389d84b8787efee51cc355cff; no voters:
I20260810 18:15:25.976719 20027 raft_consensus.cc:2781] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:25.976954 20027 raft_consensus.cc:686] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 1 LEADER]: Becoming Leader. State: Replica: 9d5574e389d84b8787efee51cc355cff, State: Running, Role: LEADER
I20260810 18:15:25.977262 20027 consensus_queue.cc:237] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } }
I20260810 18:15:25.980257 19251 catalog_manager.cc:5495] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d5574e389d84b8787efee51cc355cff (127.23.65.66). New cstate: current_term: 1 leader_uuid: "9d5574e389d84b8787efee51cc355cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d5574e389d84b8787efee51cc355cff" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34121 } health_report { overall_health: HEALTHY } } }
W20260810 18:15:26.003381 19983 tablet.cc:2369] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:15:26.036077 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.032519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.23.65.66:34121)
0810 18:15:26.032680 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:26.032684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:26.032697 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:15:26.032774 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:26.032777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:26.032834 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:26.032921 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:15:26.033114 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.033613 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.034481 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.034765 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.034857 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.035171 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:15:26.035697 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:15:26.035803 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:26.035872 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 18:15:26.035931 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 18:15:26.038033 19767 tablet.cc:2369] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:15:26.042218 19901 raft_consensus.cc:1261] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [term 1 FOLLOWER]: Refusing update from remote peer 9d5574e389d84b8787efee51cc355cff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:26.042218 19493 raft_consensus.cc:1261] T af4a79e68ca44c23bc763b3bda008b6d P 25901419426349efae35591febf1354c [term 1 FOLLOWER]: Refusing update from remote peer 9d5574e389d84b8787efee51cc355cff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:26.042949 20033 consensus_queue.cc:1035] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25901419426349efae35591febf1354c" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:26.043150 20027 consensus_queue.cc:1035] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd611055d5a64e098604e5afa07af491" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:26.051714 20038 mvcc.cc:205] Tried to move back new op lower bound from 7317035933861601280 to 7317035933605240832. Current Snapshot: MvccSnapshot[applied={T|T < 7317035933861601280}]
I20260810 18:15:26.051904 20037 mvcc.cc:205] Tried to move back new op lower bound from 7317035933861601280 to 7317035933605240832. Current Snapshot: MvccSnapshot[applied={T|T < 7317035933861601280}]
I20260810 18:15:26.059628 19251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:26.059779 19251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:26.061934 19251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b1aa70006b04d49b0c893fb705a3246: Sending DeleteTablet for 3 replicas of tablet af4a79e68ca44c23bc763b3bda008b6d
I20260810 18:15:26.062731 19877 tablet_service.cc:1514] Processing DeleteTablet for tablet af4a79e68ca44c23bc763b3bda008b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:26 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:50102
I20260810 18:15:26.062738 19458 tablet_service.cc:1514] Processing DeleteTablet for tablet af4a79e68ca44c23bc763b3bda008b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:26 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:47834
I20260810 18:15:26.063288 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 19346
I20260810 18:15:26.063690 19668 tablet_service.cc:1514] Processing DeleteTablet for tablet af4a79e68ca44c23bc763b3bda008b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:26 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:37938
I20260810 18:15:26.066404 20048 tablet_replica.cc:330] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: stopping tablet replica
I20260810 18:15:26.066533 20054 tablet_replica.cc:330] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: stopping tablet replica
I20260810 18:15:26.066831 20054 raft_consensus.cc:2229] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:15:26.066833 20048 raft_consensus.cc:2229] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:15:26.067227 20054 raft_consensus.cc:2258] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:26.067240 20048 raft_consensus.cc:2258] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:26.070663 20048 ts_tablet_manager.cc:1905] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:26.070686 20054 ts_tablet_manager.cc:1905] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:26.073418 20048 ts_tablet_manager.cc:1918] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:26.073586 20048 log.cc:1196] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/af4a79e68ca44c23bc763b3bda008b6d
I20260810 18:15:26.073689 20054 ts_tablet_manager.cc:1918] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:26.073787 20054 log.cc:1196] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/af4a79e68ca44c23bc763b3bda008b6d
I20260810 18:15:26.074075 20054 ts_tablet_manager.cc:1939] T af4a79e68ca44c23bc763b3bda008b6d P 9d5574e389d84b8787efee51cc355cff: Deleting consensus metadata
I20260810 18:15:26.074079 20048 ts_tablet_manager.cc:1939] T af4a79e68ca44c23bc763b3bda008b6d P dd611055d5a64e098604e5afa07af491: Deleting consensus metadata
I20260810 18:15:26.075266 19232 catalog_manager.cc:4841] TS 9d5574e389d84b8787efee51cc355cff (127.23.65.66:34121): tablet af4a79e68ca44c23bc763b3bda008b6d (table test-table [id=9a54d231e81a4223a8cefa8faaaffe54]) successfully deleted
I20260810 18:15:26.075384 19230 catalog_manager.cc:4841] TS dd611055d5a64e098604e5afa07af491 (127.23.65.67:43283): tablet af4a79e68ca44c23bc763b3bda008b6d (table test-table [id=9a54d231e81a4223a8cefa8faaaffe54]) successfully deleted
W20260810 18:15:26.077142 19232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:37517 (error 108)
I20260810 18:15:26.077953 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 19568
I20260810 18:15:26.078227 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.077885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37517 (local address 127.0.0.1:47840)
0810 18:15:26.078029 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:26.078103 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 18:15:26.090049 19232 catalog_manager.cc:4584] TS 25901419426349efae35591febf1354c (127.23.65.65:37517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af4a79e68ca44c23bc763b3bda008b6d: Network error: Client connection negotiation failed: client connection to 127.23.65.65:37517: connect: Connection refused (error 111)
I20260810 18:15:26.100716 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 19772
I20260810 18:15:26.114053 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 19202
2026-08-10T18:15:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2548 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T18:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:26Z Disabled control of system clock
I20260810 18:15:26.150080 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43533
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:15:26.260600 20060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:26.260878 20060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:26.260911 20060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:26.264889 20060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:26.264947 20060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:26.264961 20060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:26.264977 20060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:26.264991 20060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:26.269646 20060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:43533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:26.270654 20060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:26.272037 20060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:26.280344 20070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:26.280345 20071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:26.280337 20073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:26.280840 20060 server_base.cc:1034] running on GCE node
I20260810 18:15:26.281415 20060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:26.282496 20060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:26.283638 20060 hybrid_clock.cc:648] HybridClock initialized: now 1786385726283622 us; error 29 us; skew 500 ppm
I20260810 18:15:26.285478 20060 webserver.cc:460] Webserver started at http://127.23.65.126:35025/ using document root <none> and password file <none>
I20260810 18:15:26.286026 20060 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:26.286087 20060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:26.286291 20060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:26.288371 20060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "3978200e50934bedbfcd606147d3178b"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "d49b6ce64d3aa4857c651c1ca0f0bbb8"
server_key_iv: "07e02cb8ce80d4fb755b7cd3c07ea363"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.288928 20060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "3978200e50934bedbfcd606147d3178b"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "d49b6ce64d3aa4857c651c1ca0f0bbb8"
server_key_iv: "07e02cb8ce80d4fb755b7cd3c07ea363"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.292619 20060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:26.295023 20080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.296267 20060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:26.296382 20060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "3978200e50934bedbfcd606147d3178b"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "d49b6ce64d3aa4857c651c1ca0f0bbb8"
server_key_iv: "07e02cb8ce80d4fb755b7cd3c07ea363"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.296482 20060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:26.308221 20060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:26.308885 20060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:26.309049 20060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:26.315724 20060 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:43533
I20260810 18:15:26.315747 20158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:43533 every 8 connection(s)
I20260810 18:15:26.316614 20060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:26.319592 20161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:26.325173 20161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b: Bootstrap starting.
I20260810 18:15:26.325479 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 20060
I20260810 18:15:26.325603 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:26.325860 23813 external_mini_cluster.cc:1428] Setting key feb146cc67108eaf564f36368ada9192
I20260810 18:15:26.327739 20161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:26.328464 20161 log.cc:824] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:26.330287 20161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b: No bootstrap required, opened a new log
I20260810 18:15:26.333676 20161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3978200e50934bedbfcd606147d3178b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43533 } }
I20260810 18:15:26.333941 20161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:26.333998 20161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3978200e50934bedbfcd606147d3178b, State: Initialized, Role: FOLLOWER
I20260810 18:15:26.334412 20161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3978200e50934bedbfcd606147d3178b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43533 } }
I20260810 18:15:26.334532 20161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:26.334589 20161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:26.334648 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.327158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.23.65.126:43533)
0810 18:15:26.327793 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:26.327813 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:26.327867 (+ 54us) server_negotiation.cc:407] Connection header received
0810 18:15:26.328548 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:26.328572 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:26.328965 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:26.329273 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:26.329694 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.330660 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.331506 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.332016 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.332161 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.332927 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:26.332932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:26.332944 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:26.333319 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:26.333374 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:26.333395 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:26.333481 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:26.333606 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 18:15:26.333849 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
I20260810 18:15:26.334686 20161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:26.335490 20161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3978200e50934bedbfcd606147d3178b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43533 } }
I20260810 18:15:26.335834 20161 leader_election.cc:302] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3978200e50934bedbfcd606147d3178b; no voters:
I20260810 18:15:26.336102 20161 leader_election.cc:288] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:26.336186 20167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:26.336395 20167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [term 1 LEADER]: Becoming Leader. State: Replica: 3978200e50934bedbfcd606147d3178b, State: Running, Role: LEADER
I20260810 18:15:26.336689 20167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3978200e50934bedbfcd606147d3178b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43533 } }
I20260810 18:15:26.337352 20161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:26.337787 20171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3978200e50934bedbfcd606147d3178b. Latest consensus state: current_term: 1 leader_uuid: "3978200e50934bedbfcd606147d3178b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3978200e50934bedbfcd606147d3178b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43533 } } }
I20260810 18:15:26.337930 20171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:26.338348 20175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:26.338567 20172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3978200e50934bedbfcd606147d3178b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3978200e50934bedbfcd606147d3178b" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43533 } } }
I20260810 18:15:26.338689 20172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:26.341832 20175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:26.347297 20175 catalog_manager.cc:1347] Generated new cluster ID: 62612134802c494aa13c2a46be81df66
I20260810 18:15:26.347366 20175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:26.355268 20175 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:26.356387 20175 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:26.363386 20175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b: Generated new TSK 0
I20260810 18:15:26.364104 20175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:26.375662 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43533
--builtin_ntp_servers=127.23.65.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:15:26.487754 20195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:26.488001 20195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:26.488039 20195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:26.491993 20195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:26.492041 20195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:26.492101 20195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:15:26.496734 20195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:26.497721 20195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:26.499060 20195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:26.508368 20200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:26.508445 20208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:26.508656 20195 server_base.cc:1034] running on GCE node
W20260810 18:15:26.508364 20202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:26.509105 20195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:26.509721 20195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:26.510878 20195 hybrid_clock.cc:648] HybridClock initialized: now 1786385726510861 us; error 28 us; skew 500 ppm
I20260810 18:15:26.512655 20195 webserver.cc:460] Webserver started at http://127.23.65.65:38421/ using document root <none> and password file <none>
I20260810 18:15:26.513195 20195 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:26.513270 20195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:26.513477 20195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:26.515465 20195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "b6673223712342109697de7de722eff0"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "f68f5f5e18be35d728b000b8863a5f2a"
server_key_iv: "c3c55761bc0ebdbc2c531491892ec152"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.516044 20195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "b6673223712342109697de7de722eff0"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "f68f5f5e18be35d728b000b8863a5f2a"
server_key_iv: "c3c55761bc0ebdbc2c531491892ec152"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.519527 20195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:15:26.521934 20220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.523428 20195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:15:26.523590 20195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "b6673223712342109697de7de722eff0"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "f68f5f5e18be35d728b000b8863a5f2a"
server_key_iv: "c3c55761bc0ebdbc2c531491892ec152"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.523699 20195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:26.537691 20195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:26.538365 20195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:26.538553 20195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:26.539157 20195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:26.540158 20195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:26.540210 20195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.540271 20195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:26.540299 20195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.548906 20195 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:42939
I20260810 18:15:26.548939 20393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:42939 every 8 connection(s)
I20260810 18:15:26.549806 20195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:15:26.552588 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 20195
I20260810 18:15:26.552734 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:15:26.553043 23813 external_mini_cluster.cc:1428] Setting key dca5757432941ffd029a2a92ac107500
I20260810 18:15:26.555001 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43533
--builtin_ntp_servers=127.23.65.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:15:26.559222 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.551624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:36295 (local address 127.23.65.126:43533)
0810 18:15:26.551796 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:26.551800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:26.552919 (+ 1119us) server_negotiation.cc:407] Connection header received
0810 18:15:26.554419 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:26.554422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:26.554476 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:26.554562 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:26.555489 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.556222 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.557397 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.557621 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.557712 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.558752 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:26.558756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:26.558759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:26.558772 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:26.558792 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:26.558801 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:26.558858 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:26.559003 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 18:15:26.559066 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 18:15:26.560087 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.552027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43533 (local address 127.23.65.65:36295)
0810 18:15:26.552759 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:26.552819 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:26.554188 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:26.554732 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:26.554739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:26.554823 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:26.555355 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.555370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.556375 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.556378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.557271 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.557281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.557765 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.557770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.557925 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.558498 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:15:26.558499 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:26.558537 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:15:26.558597 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:26.558869 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:26.558878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:26.558884 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:15:26.558988 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 18:15:26.559253 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260810 18:15:26.561376 20394 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43533
I20260810 18:15:26.561604 20394 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:26.562117 20394 heartbeater.cc:511] Master 127.23.65.126:43533 requested a full tablet report, sending...
I20260810 18:15:26.563866 20106 ts_manager.cc:194] Registered new tserver with Master: b6673223712342109697de7de722eff0 (127.23.65.65:42939)
I20260810 18:15:26.565533 20106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.65:36295
W20260810 18:15:26.668941 20397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:26.669195 20397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:26.669231 20397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:26.673221 20397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:26.673286 20397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:26.673362 20397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:15:26.678082 20397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:26.679082 20397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:26.680452 20397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:26.689682 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:26.689787 20397 server_base.cc:1034] running on GCE node
W20260810 18:15:26.689692 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:26.689723 20405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:26.690296 20397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:26.690876 20397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:26.692029 20397 hybrid_clock.cc:648] HybridClock initialized: now 1786385726691994 us; error 46 us; skew 500 ppm
I20260810 18:15:26.693755 20397 webserver.cc:460] Webserver started at http://127.23.65.66:41051/ using document root <none> and password file <none>
I20260810 18:15:26.694299 20397 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:26.694366 20397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:26.694566 20397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:26.696615 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "a06b3ca0b75e4845a42a6807f71b8eab"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "1a4bdf0a8d4e4ccb9efe03de27d5180f"
server_key_iv: "97b90048dd28a16b82eacda9e7690c76"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.697152 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "a06b3ca0b75e4845a42a6807f71b8eab"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "1a4bdf0a8d4e4ccb9efe03de27d5180f"
server_key_iv: "97b90048dd28a16b82eacda9e7690c76"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.700749 20397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:26.703055 20412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.704246 20397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:26.704350 20397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "a06b3ca0b75e4845a42a6807f71b8eab"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "1a4bdf0a8d4e4ccb9efe03de27d5180f"
server_key_iv: "97b90048dd28a16b82eacda9e7690c76"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.704443 20397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:26.725250 20397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:26.725940 20397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:26.726114 20397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:26.726758 20397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:26.727770 20397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:26.727823 20397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.727876 20397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:26.727901 20397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.735885 20397 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:38745
I20260810 18:15:26.735952 20592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:38745 every 8 connection(s)
I20260810 18:15:26.736739 20397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:15:26.741947 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 20397
I20260810 18:15:26.742075 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:15:26.742400 23813 external_mini_cluster.cc:1428] Setting key 3061f520a76466e1b4d429f40dff3225
I20260810 18:15:26.744411 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43533
--builtin_ntp_servers=127.23.65.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:15:26.746433 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.738441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:33127 (local address 127.23.65.126:43533)
0810 18:15:26.738586 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:26.738590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:26.739904 (+ 1314us) server_negotiation.cc:407] Connection header received
0810 18:15:26.741249 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:26.741252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:26.741300 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:26.741368 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:26.742400 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.743042 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.744479 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.744770 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.744847 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.745958 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:26.745961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:26.745963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:26.745975 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:26.745995 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:26.746002 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:26.746052 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:26.746215 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 18:15:26.746292 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:15:26.747309 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.738890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43533 (local address 127.23.65.66:33127)
0810 18:15:26.739748 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:26.739803 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:26.741071 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:26.741499 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:26.741506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:26.741584 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:26.742262 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.742284 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.743163 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.743167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.744364 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.744375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.744916 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.744920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.745105 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.745689 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:15:26.745691 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:26.745743 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:15:26.745822 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:26.746068 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:26.746076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:26.746081 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:15:26.746207 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 18:15:26.746467 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":141,"threads_started":1}
I20260810 18:15:26.748353 20598 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43533
I20260810 18:15:26.748587 20598 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:26.749203 20598 heartbeater.cc:511] Master 127.23.65.126:43533 requested a full tablet report, sending...
I20260810 18:15:26.750312 20106 ts_manager.cc:194] Registered new tserver with Master: a06b3ca0b75e4845a42a6807f71b8eab (127.23.65.66:38745)
I20260810 18:15:26.750856 20106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.66:33127
W20260810 18:15:26.855372 20602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:26.855667 20602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:26.855720 20602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:26.859619 20602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:26.859668 20602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:26.859726 20602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:15:26.864401 20602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:26.865455 20602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:26.866803 20602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:15:26.875859 20602 server_base.cc:1034] running on GCE node
W20260810 18:15:26.875828 20611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:26.875819 20613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:26.875996 20610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:26.876443 20602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:26.877126 20602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:26.878304 20602 hybrid_clock.cc:648] HybridClock initialized: now 1786385726878273 us; error 46 us; skew 500 ppm
I20260810 18:15:26.880064 20602 webserver.cc:460] Webserver started at http://127.23.65.67:40329/ using document root <none> and password file <none>
I20260810 18:15:26.880612 20602 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:26.880689 20602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:26.880893 20602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:26.882864 20602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "69f2c012636a4d41b03643cb0fdebea2"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "aee8bcc5bb51d333a8eae202ea3d8380"
server_key_iv: "4dfdabb5df44026713543806a0e8cd45"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.883384 20602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "69f2c012636a4d41b03643cb0fdebea2"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "aee8bcc5bb51d333a8eae202ea3d8380"
server_key_iv: "4dfdabb5df44026713543806a0e8cd45"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.887053 20602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:15:26.889442 20620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.890609 20602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:26.890730 20602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "69f2c012636a4d41b03643cb0fdebea2"
format_stamp: "Formatted at 2026-08-10 18:15:26 on dist-test-slave-jpj1"
server_key: "aee8bcc5bb51d333a8eae202ea3d8380"
server_key_iv: "4dfdabb5df44026713543806a0e8cd45"
server_key_version: "encryptionkey@0"
I20260810 18:15:26.890829 20602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:26.900876 20602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:26.901494 20602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:26.901651 20602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:26.902190 20602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:26.903105 20602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:26.903152 20602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.903204 20602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:26.903230 20602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:26.911656 20602 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:43801
I20260810 18:15:26.911697 20792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:43801 every 8 connection(s)
I20260810 18:15:26.912537 20602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:15:26.921195 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.914310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:36911 (local address 127.23.65.126:43533)
0810 18:15:26.914451 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:26.914456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:26.915516 (+ 1060us) server_negotiation.cc:407] Connection header received
0810 18:15:26.916913 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:26.916916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:26.916962 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:26.917035 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:26.917917 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.918432 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.919509 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.919773 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.919851 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.920788 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:26.920791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:26.920793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:26.920806 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:26.920824 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:26.920831 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:26.920883 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:26.921016 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 18:15:26.921069 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:15:26.921324 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 20602
I20260810 18:15:26.921455 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:15:26.921769 23813 external_mini_cluster.cc:1428] Setting key 84c296ef917bf91982c0c828c017a9aa
I20260810 18:15:26.922266 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.914730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43533 (local address 127.23.65.67:36911)
0810 18:15:26.915377 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:26.915417 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:26.916724 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:26.917220 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:26.917226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:26.917319 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:26.917800 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.917814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.918525 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.918527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.919418 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.919428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.919896 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.919899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.920039 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.920559 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:15:26.920561 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:26.920598 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:15:26.920658 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:26.920889 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:26.920893 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:26.920896 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:15:26.920981 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:15:26.921247 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260810 18:15:26.923261 20794 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43533
I20260810 18:15:26.923468 20794 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:26.924019 20794 heartbeater.cc:511] Master 127.23.65.126:43533 requested a full tablet report, sending...
I20260810 18:15:26.924959 20106 ts_manager.cc:194] Registered new tserver with Master: 69f2c012636a4d41b03643cb0fdebea2 (127.23.65.67:43801)
I20260810 18:15:26.925499 20106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.67:36911
I20260810 18:15:26.935416 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:15:26.940925 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.937276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.23.65.126:43533)
0810 18:15:26.937385 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:26.937390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:26.937577 (+ 187us) server_negotiation.cc:407] Connection header received
0810 18:15:26.937706 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:26.937710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:26.937749 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:26.937809 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:26.938137 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.938669 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.939520 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.939767 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.939844 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.940556 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:26.940559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:26.940561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:26.940571 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:26.940587 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:26.940594 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:26.940642 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:26.940742 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 18:15:26.940791 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 18:15:26.943564 20106 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:15:26.946574 20106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:15:26.960255 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.955176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.23.65.65:42939)
0810 18:15:26.955714 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:26.955726 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:26.955754 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:15:26.955994 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:26.955999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:26.956192 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:26.956339 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:26.956696 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.957431 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.959027 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.959615 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.959712 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.959920 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:26.960011 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 18:15:26.960103 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":128,"threads_started":1}
I20260810 18:15:26.960259 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.954700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:42939 (local address 127.0.0.1:40212)
0810 18:15:26.955571 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:26.955583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:26.955660 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:26.956490 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:26.956492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:26.956508 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:26.956590 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.956596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.957542 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.957545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.958885 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.958896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.959724 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.959727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.959835 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.959872 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 18:15:26.959983 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 18:15:26.960104 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":6,"thread_start_us":93,"threads_started":1}
I20260810 18:15:26.960328 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.954476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:43801 (local address 127.0.0.1:51202)
0810 18:15:26.955167 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:26.955204 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:26.955337 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:26.956332 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:26.956335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:26.956354 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:26.956458 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.956465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.957855 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.957859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.958888 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.958897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.959940 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.959944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.960030 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.960053 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:15:26.960103 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 18:15:26.960185 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":22,"spinlock_wait_cycles":5120,"thread_start_us":105,"threads_started":1}
I20260810 18:15:26.960510 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.954362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:38745 (local address 127.0.0.1:43642)
0810 18:15:26.955152 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:26.955205 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:26.955337 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:26.956168 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:26.956174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:26.956196 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:26.956338 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.956346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.957800 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.957804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.958974 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:26.958982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.960168 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.960172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:26.960256 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.960276 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:15:26.960338 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:15:26.960399 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":38,"spinlock_wait_cycles":6400,"thread_start_us":72,"threads_started":1}
I20260810 18:15:26.960988 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.955175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.23.65.67:43801)
0810 18:15:26.955910 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:26.955922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:26.955944 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:15:26.956020 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:26.956025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:26.956201 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:26.956342 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:26.956581 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.957715 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.959000 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.959802 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.960494 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.960678 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:26.960739 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 18:15:26.960828 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":126,"threads_started":1}
I20260810 18:15:26.960983 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:26.954506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.23.65.66:38745)
0810 18:15:26.954910 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:26.954921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:26.955479 (+ 558us) server_negotiation.cc:407] Connection header received
0810 18:15:26.955593 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:26.955598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:26.955823 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:26.955977 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:26.956467 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.957682 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.959079 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:26.959842 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:26.959931 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:26.960110 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:26.960741 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 18:15:26.960828 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":91,"threads_started":1}
I20260810 18:15:26.963744 20691 tablet_service.cc:1467] Processing CreateTablet for tablet 89ded71257414f8c92e5b4bcdad0c484 (DEFAULT_TABLE table=test-table [id=f4e87b8d76a1424bb28659026eef2a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:26.963899 20496 tablet_service.cc:1467] Processing CreateTablet for tablet 89ded71257414f8c92e5b4bcdad0c484 (DEFAULT_TABLE table=test-table [id=f4e87b8d76a1424bb28659026eef2a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:26.963793 20290 tablet_service.cc:1467] Processing CreateTablet for tablet 89ded71257414f8c92e5b4bcdad0c484 (DEFAULT_TABLE table=test-table [id=f4e87b8d76a1424bb28659026eef2a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:26.964695 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ded71257414f8c92e5b4bcdad0c484. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:26.964807 20290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ded71257414f8c92e5b4bcdad0c484. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:26.964807 20496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ded71257414f8c92e5b4bcdad0c484. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:26.970814 20813 tablet_bootstrap.cc:492] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: Bootstrap starting.
I20260810 18:15:26.971936 20816 tablet_bootstrap.cc:492] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: Bootstrap starting.
I20260810 18:15:26.972645 20814 tablet_bootstrap.cc:492] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0: Bootstrap starting.
I20260810 18:15:26.973206 20813 tablet_bootstrap.cc:654] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:26.974038 20816 tablet_bootstrap.cc:654] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:26.974069 20813 log.cc:824] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:26.974398 20814 tablet_bootstrap.cc:654] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:26.974782 20816 log.cc:824] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:26.975203 20814 log.cc:824] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:26.976253 20813 tablet_bootstrap.cc:492] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: No bootstrap required, opened a new log
I20260810 18:15:26.976459 20813 ts_tablet_manager.cc:1397] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:15:26.976879 20816 tablet_bootstrap.cc:492] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: No bootstrap required, opened a new log
I20260810 18:15:26.977061 20816 ts_tablet_manager.cc:1397] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:15:26.977609 20814 tablet_bootstrap.cc:492] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0: No bootstrap required, opened a new log
I20260810 18:15:26.977815 20814 ts_tablet_manager.cc:1397] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 18:15:26.980151 20816 raft_consensus.cc:348] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } }
I20260810 18:15:26.980365 20816 raft_consensus.cc:374] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:26.980417 20816 raft_consensus.cc:729] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06b3ca0b75e4845a42a6807f71b8eab, State: Initialized, Role: FOLLOWER
I20260810 18:15:26.980821 20816 consensus_queue.cc:260] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } }
I20260810 18:15:26.980703 20813 raft_consensus.cc:348] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } }
I20260810 18:15:26.980952 20813 raft_consensus.cc:374] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:26.981007 20813 raft_consensus.cc:729] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f2c012636a4d41b03643cb0fdebea2, State: Initialized, Role: FOLLOWER
I20260810 18:15:26.981442 20598 heartbeater.cc:503] Master 127.23.65.126:43533 was elected leader, sending a full tablet report...
I20260810 18:15:26.981438 20813 consensus_queue.cc:260] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } }
I20260810 18:15:26.981566 20814 raft_consensus.cc:348] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } }
I20260810 18:15:26.981812 20816 ts_tablet_manager.cc:1428] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:15:26.981813 20814 raft_consensus.cc:374] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:26.981870 20814 raft_consensus.cc:729] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6673223712342109697de7de722eff0, State: Initialized, Role: FOLLOWER
I20260810 18:15:26.982246 20814 consensus_queue.cc:260] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } }
I20260810 18:15:26.982379 20813 ts_tablet_manager.cc:1428] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:15:26.982692 20794 heartbeater.cc:503] Master 127.23.65.126:43533 was elected leader, sending a full tablet report...
I20260810 18:15:26.983116 20394 heartbeater.cc:503] Master 127.23.65.126:43533 was elected leader, sending a full tablet report...
I20260810 18:15:26.983150 20814 ts_tablet_manager.cc:1428] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
W20260810 18:15:26.987811 20599 tablet.cc:2369] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:15:27.050941 20395 tablet.cc:2369] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:15:27.163614 20795 tablet.cc:2369] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:15:27.195183 20821 raft_consensus.cc:482] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:27.195426 20821 raft_consensus.cc:504] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } }
I20260810 18:15:27.196682 20821 leader_election.cc:288] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a06b3ca0b75e4845a42a6807f71b8eab (127.23.65.66:38745), b6673223712342109697de7de722eff0 (127.23.65.65:42939)
I20260810 18:15:27.200352 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:27.196939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:42939 (local address 127.23.65.67:49271)
0810 18:15:27.197185 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:27.197200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:27.197318 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:27.197565 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:27.197568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:27.197586 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:27.197698 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:27.197704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.198422 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.198425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:27.199212 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:27.199221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.199954 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.199958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:27.200057 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:27.200079 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:15:27.200144 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:15:27.200214 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260810 18:15:27.200352 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:27.197043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:49271 (local address 127.23.65.65:42939)
0810 18:15:27.197196 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:27.197201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:27.197280 (+ 79us) server_negotiation.cc:407] Connection header received
0810 18:15:27.197416 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:27.197418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:27.197465 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:27.197539 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:27.197811 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.198319 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.199340 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.199846 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.199954 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:27.200008 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:27.200170 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 18:15:27.200241 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:15:27.200572 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:27.197043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:46325 (local address 127.23.65.66:38745)
0810 18:15:27.197184 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:27.197189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:27.197505 (+ 316us) server_negotiation.cc:407] Connection header received
0810 18:15:27.197601 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:27.197604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:27.197650 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:27.197710 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:27.197981 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.198464 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.199457 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.200049 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.200157 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:27.200222 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:27.200386 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 18:15:27.200469 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:15:27.200569 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:27.196939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:38745 (local address 127.23.65.67:46325)
0810 18:15:27.197409 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:27.197423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:27.197515 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:27.197732 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:27.197735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:27.197751 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:27.197873 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:27.197879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.198575 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.198578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:27.199346 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:27.199358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.200157 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.200160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:27.200266 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:27.200290 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:15:27.200358 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:15:27.200425 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":50,"thread_start_us":148,"threads_started":1}
I20260810 18:15:27.201098 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ded71257414f8c92e5b4bcdad0c484" candidate_uuid: "69f2c012636a4d41b03643cb0fdebea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6673223712342109697de7de722eff0" is_pre_election: true
I20260810 18:15:27.201236 20532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ded71257414f8c92e5b4bcdad0c484" candidate_uuid: "69f2c012636a4d41b03643cb0fdebea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" is_pre_election: true
I20260810 18:15:27.201422 20319 raft_consensus.cc:2454] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f2c012636a4d41b03643cb0fdebea2 in term 0.
I20260810 18:15:27.201529 20532 raft_consensus.cc:2454] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f2c012636a4d41b03643cb0fdebea2 in term 0.
I20260810 18:15:27.201959 20622 leader_election.cc:302] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 69f2c012636a4d41b03643cb0fdebea2, a06b3ca0b75e4845a42a6807f71b8eab, b6673223712342109697de7de722eff0; no voters:
I20260810 18:15:27.202224 20821 raft_consensus.cc:2781] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:15:27.202317 20821 raft_consensus.cc:482] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:15:27.202366 20821 raft_consensus.cc:3037] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:27.203429 20821 raft_consensus.cc:504] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } }
I20260810 18:15:27.203789 20821 leader_election.cc:288] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [CANDIDATE]: Term 1 election: Requested vote from peers a06b3ca0b75e4845a42a6807f71b8eab (127.23.65.66:38745), b6673223712342109697de7de722eff0 (127.23.65.65:42939)
I20260810 18:15:27.204104 20532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ded71257414f8c92e5b4bcdad0c484" candidate_uuid: "69f2c012636a4d41b03643cb0fdebea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06b3ca0b75e4845a42a6807f71b8eab"
I20260810 18:15:27.204103 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ded71257414f8c92e5b4bcdad0c484" candidate_uuid: "69f2c012636a4d41b03643cb0fdebea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6673223712342109697de7de722eff0"
I20260810 18:15:27.204211 20319 raft_consensus.cc:3037] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:27.204221 20532 raft_consensus.cc:3037] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:27.205214 20319 raft_consensus.cc:2454] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f2c012636a4d41b03643cb0fdebea2 in term 1.
I20260810 18:15:27.205226 20532 raft_consensus.cc:2454] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f2c012636a4d41b03643cb0fdebea2 in term 1.
I20260810 18:15:27.205497 20625 leader_election.cc:302] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69f2c012636a4d41b03643cb0fdebea2, a06b3ca0b75e4845a42a6807f71b8eab; no voters:
I20260810 18:15:27.205644 20821 raft_consensus.cc:2781] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:27.205844 20821 raft_consensus.cc:686] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 1 LEADER]: Becoming Leader. State: Replica: 69f2c012636a4d41b03643cb0fdebea2, State: Running, Role: LEADER
I20260810 18:15:27.206178 20821 consensus_queue.cc:237] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } }
I20260810 18:15:27.209203 20103 catalog_manager.cc:5495] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f2c012636a4d41b03643cb0fdebea2 (127.23.65.67). New cstate: current_term: 1 leader_uuid: "69f2c012636a4d41b03643cb0fdebea2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f2c012636a4d41b03643cb0fdebea2" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 43801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 } health_report { overall_health: UNKNOWN } } }
I20260810 18:15:27.251134 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:27.248115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.23.65.67:43801)
0810 18:15:27.248233 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:27.248237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:27.248250 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:15:27.248328 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:27.248330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:27.248387 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:27.248473 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:15:27.248657 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.249162 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.249948 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:27.250148 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:27.250211 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:27.250437 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:15:27.250846 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:15:27.250931 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:27.250962 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 18:15:27.251005 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:15:27.256925 20532 raft_consensus.cc:1261] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [term 1 FOLLOWER]: Refusing update from remote peer 69f2c012636a4d41b03643cb0fdebea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:27.256925 20319 raft_consensus.cc:1261] T 89ded71257414f8c92e5b4bcdad0c484 P b6673223712342109697de7de722eff0 [term 1 FOLLOWER]: Refusing update from remote peer 69f2c012636a4d41b03643cb0fdebea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:27.257613 20824 consensus_queue.cc:1035] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6673223712342109697de7de722eff0" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:27.257797 20821 consensus_queue.cc:1035] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a06b3ca0b75e4845a42a6807f71b8eab" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:27.264361 20837 mvcc.cc:205] Tried to move back new op lower bound from 7317035938836852736 to 7317035938638786560. Current Snapshot: MvccSnapshot[applied={T|T < 7317035938836852736}]
I20260810 18:15:27.272457 20103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:27.272608 20103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:27.274914 20103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3978200e50934bedbfcd606147d3178b: Sending DeleteTablet for 3 replicas of tablet 89ded71257414f8c92e5b4bcdad0c484
I20260810 18:15:27.275715 20496 tablet_service.cc:1514] Processing DeleteTablet for tablet 89ded71257414f8c92e5b4bcdad0c484 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:27 UTC) from {username='slave'} at 127.0.0.1:43642
I20260810 18:15:27.275717 20691 tablet_service.cc:1514] Processing DeleteTablet for tablet 89ded71257414f8c92e5b4bcdad0c484 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:27 UTC) from {username='slave'} at 127.0.0.1:51202
I20260810 18:15:27.276126 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 20195
I20260810 18:15:27.276470 20848 tablet_replica.cc:330] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: stopping tablet replica
I20260810 18:15:27.276690 20847 tablet_replica.cc:330] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: stopping tablet replica
I20260810 18:15:27.277139 20848 raft_consensus.cc:2229] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:15:27.277575 20847 raft_consensus.cc:2229] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:15:27.278071 20847 raft_consensus.cc:2258] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:27.278501 20848 raft_consensus.cc:2258] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:27.288749 20847 ts_tablet_manager.cc:1905] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:27.288944 20848 ts_tablet_manager.cc:1905] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:27.291463 20847 ts_tablet_manager.cc:1918] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:27.291594 20847 log.cc:1196] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/89ded71257414f8c92e5b4bcdad0c484
I20260810 18:15:27.291767 20848 ts_tablet_manager.cc:1918] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:27.291869 20848 log.cc:1196] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/89ded71257414f8c92e5b4bcdad0c484
I20260810 18:15:27.291889 20847 ts_tablet_manager.cc:1939] T 89ded71257414f8c92e5b4bcdad0c484 P 69f2c012636a4d41b03643cb0fdebea2: Deleting consensus metadata
I20260810 18:15:27.292192 20848 ts_tablet_manager.cc:1939] T 89ded71257414f8c92e5b4bcdad0c484 P a06b3ca0b75e4845a42a6807f71b8eab: Deleting consensus metadata
W20260810 18:15:27.292609 20083 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:42939 (error 108)
I20260810 18:15:27.293205 20084 catalog_manager.cc:4841] TS 69f2c012636a4d41b03643cb0fdebea2 (127.23.65.67:43801): tablet 89ded71257414f8c92e5b4bcdad0c484 (table test-table [id=f4e87b8d76a1424bb28659026eef2a50]) successfully deleted
I20260810 18:15:27.293335 20086 catalog_manager.cc:4841] TS a06b3ca0b75e4845a42a6807f71b8eab (127.23.65.66:38745): tablet 89ded71257414f8c92e5b4bcdad0c484 (table test-table [id=f4e87b8d76a1424bb28659026eef2a50]) successfully deleted
I20260810 18:15:27.293758 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 20397
I20260810 18:15:27.297848 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:27.293860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:42939 (local address 127.0.0.1:40224)
0810 18:15:27.297672 (+ 3812us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:27.297754 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3717}
W20260810 18:15:27.298060 20083 catalog_manager.cc:4584] TS b6673223712342109697de7de722eff0 (127.23.65.65:42939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89ded71257414f8c92e5b4bcdad0c484: Network error: Client connection negotiation failed: client connection to 127.23.65.65:42939: connect: Connection refused (error 111)
I20260810 18:15:27.308514 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 20602
I20260810 18:15:27.321967 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 20060
2026-08-10T18:15:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1209 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:27 dist-test-slave-jpj1 krb5kdc[20852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:27 dist-test-slave-jpj1 krb5kdc[20852](info): set up 2 sockets
Aug 10 18:15:27 dist-test-slave-jpj1 krb5kdc[20852](info): commencing operation
krb5kdc: starting...
W20260810 18:15:28.484180 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:28 dist-test-slave-jpj1 krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:15:28.604008 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:39755
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:28.714866 20871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:28.715121 20871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:28.715178 20871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:28.719131 20871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:28.719187 20871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:28.719213 20871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:28.719235 20871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:28.719264 20871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:28.724395 20871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:28.725409 20871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:28.726398 20871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:28.734706 20882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:28.734709 20889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:28.734784 20871 server_base.cc:1034] running on GCE node
W20260810 18:15:28.734733 20879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:28.735652 20871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:28.736717 20871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:28.737883 20871 hybrid_clock.cc:648] HybridClock initialized: now 1786385728737867 us; error 29 us; skew 500 ppm
Aug 10 18:15:28 dist-test-slave-jpj1 krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385728, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:28.740607 20871 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:15:28.741482 20871 webserver.cc:460] Webserver started at http://127.23.65.126:33669/ using document root <none> and password file <none>
I20260810 18:15:28.742010 20871 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:28.742079 20871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:28.742276 20871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:28.744336 20871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "fa5e6d165bc441fe9ff454d47dbe45f4"
format_stamp: "Formatted at 2026-08-10 18:15:28 on dist-test-slave-jpj1"
server_key: "d8b78da8c6bbabb7360448d16f721000"
server_key_iv: "2f84a8f3f76bb318d77907af9734a28f"
server_key_version: "encryptionkey@0"
I20260810 18:15:28.744870 20871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "fa5e6d165bc441fe9ff454d47dbe45f4"
format_stamp: "Formatted at 2026-08-10 18:15:28 on dist-test-slave-jpj1"
server_key: "d8b78da8c6bbabb7360448d16f721000"
server_key_iv: "2f84a8f3f76bb318d77907af9734a28f"
server_key_version: "encryptionkey@0"
I20260810 18:15:28.748485 20871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:28.750763 20900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:28.751957 20871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:28.752071 20871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "fa5e6d165bc441fe9ff454d47dbe45f4"
format_stamp: "Formatted at 2026-08-10 18:15:28 on dist-test-slave-jpj1"
server_key: "d8b78da8c6bbabb7360448d16f721000"
server_key_iv: "2f84a8f3f76bb318d77907af9734a28f"
server_key_version: "encryptionkey@0"
I20260810 18:15:28.752164 20871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:28.769479 20871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:28.772540 20871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:28.772719 20871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:28.779663 20871 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:39755
I20260810 18:15:28.779718 20983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:39755 every 8 connection(s)
I20260810 18:15:28.780591 20871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:28.783177 20984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:28.788523 20984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4: Bootstrap starting.
I20260810 18:15:28.789515 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 20871
I20260810 18:15:28.789695 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:28.790016 23813 external_mini_cluster.cc:1428] Setting key f29da782ec91819d1c2e62fb45583a2a
I20260810 18:15:28.791359 20984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:28.792109 20984 log.cc:824] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:28.793890 20984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4: No bootstrap required, opened a new log
I20260810 18:15:28.796788 20984 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5e6d165bc441fe9ff454d47dbe45f4" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39755 } }
I20260810 18:15:28.797009 20984 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:28.797093 20984 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5e6d165bc441fe9ff454d47dbe45f4, State: Initialized, Role: FOLLOWER
I20260810 18:15:28.797566 20984 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5e6d165bc441fe9ff454d47dbe45f4" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39755 } }
I20260810 18:15:28.797734 20984 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:28.797806 20984 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:28.797910 20984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:28.798893 20984 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5e6d165bc441fe9ff454d47dbe45f4" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39755 } }
Aug 10 18:15:28 dist-test-slave-jpj1 krb5kdc[20852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:28.799214 20984 leader_election.cc:302] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa5e6d165bc441fe9ff454d47dbe45f4; no voters:
I20260810 18:15:28.799556 20984 leader_election.cc:288] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:28.799664 20993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:28.799870 20993 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [term 1 LEADER]: Becoming Leader. State: Replica: fa5e6d165bc441fe9ff454d47dbe45f4, State: Running, Role: LEADER
I20260810 18:15:28.800462 20993 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5e6d165bc441fe9ff454d47dbe45f4" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39755 } }
I20260810 18:15:28.800851 20984 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:28.802109 20995 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5e6d165bc441fe9ff454d47dbe45f4. Latest consensus state: current_term: 1 leader_uuid: "fa5e6d165bc441fe9ff454d47dbe45f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5e6d165bc441fe9ff454d47dbe45f4" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39755 } } }
I20260810 18:15:28.802122 20994 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa5e6d165bc441fe9ff454d47dbe45f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5e6d165bc441fe9ff454d47dbe45f4" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 39755 } } }
I20260810 18:15:28.802253 20994 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:28.802253 20995 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:28.805200 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:28.791111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.23.65.126:39755)
0810 18:15:28.791505 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:28.791521 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:28.791580 (+ 59us) server_negotiation.cc:407] Connection header received
0810 18:15:28.792215 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:28.792237 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:28.792580 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:28.792892 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:28.793703 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:28.794692 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:28.795562 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:28.796746 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:28.796921 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:28.799560 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:28.799568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:28.799586 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:28.799617 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:28.801595 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:28.802155 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:28.802160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:28.802166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:28.802300 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:28.802632 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:28.802635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:28.802637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:28.803164 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:28.804137 (+ 973us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:28.804287 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 18:15:28.804509 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":131,"threads_started":1}
W20260810 18:15:28.805709 21010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:15:28.805790 21010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:15:28.805860 21003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:28.806695 21003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:28.812381 21003 catalog_manager.cc:1347] Generated new cluster ID: a0596f4c51da434faff8bd7fd37733c6
I20260810 18:15:28.812438 21003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:28.836561 21003 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:28.837442 21003 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:28.844659 21003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4: Generated new TSK 0
I20260810 18:15:28.845256 21003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:15:28.879933 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:39755
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:28.990429 21021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:28.990677 21021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:28.990718 21021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:28.994717 21021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:28.994776 21021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:28.994843 21021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:15:28.999946 21021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:29.000959 21021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:29.002015 21021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:29.011555 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:29.011556 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:29.011647 21021 server_base.cc:1034] running on GCE node
W20260810 18:15:29.011556 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:29.012255 21021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:29.012850 21021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:29.014000 21021 hybrid_clock.cc:648] HybridClock initialized: now 1786385729013973 us; error 40 us; skew 500 ppm
Aug 10 18:15:29 dist-test-slave-jpj1 krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385729, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:29.016713 21021 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:15:29.017611 21021 webserver.cc:460] Webserver started at http://127.23.65.65:37451/ using document root <none> and password file <none>
I20260810 18:15:29.018141 21021 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:29.018200 21021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:29.018410 21021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:29.020520 21021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "cc53210eb47d4cc9bcc900ccff3969d3"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "bfe2ff1cd5256e2ffee11a45dd85007e"
server_key_iv: "4588e0bfcf2aa20fd428c2ada851c8b1"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.021064 21021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "cc53210eb47d4cc9bcc900ccff3969d3"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "bfe2ff1cd5256e2ffee11a45dd85007e"
server_key_iv: "4588e0bfcf2aa20fd428c2ada851c8b1"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.024896 21021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 18:15:29.027228 21047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.028405 21021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:15:29.028509 21021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "cc53210eb47d4cc9bcc900ccff3969d3"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "bfe2ff1cd5256e2ffee11a45dd85007e"
server_key_iv: "4588e0bfcf2aa20fd428c2ada851c8b1"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.028609 21021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:29.046890 21021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:29.049891 21021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:29.050068 21021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:29.050639 21021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:29.051470 21021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:29.051522 21021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.051592 21021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:29.051622 21021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.060081 21021 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:39167
I20260810 18:15:29.060109 21223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:39167 every 8 connection(s)
I20260810 18:15:29.060945 21021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:15:29.066031 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 21021
I20260810 18:15:29.066161 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:15:29.066458 23813 external_mini_cluster.cc:1428] Setting key 95c8d536ff0f4405d4cb306ff7af2a54
Aug 10 18:15:29 dist-test-slave-jpj1 krb5kdc[20852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385729, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:29.074594 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.062681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:54761 (local address 127.23.65.126:39755)
0810 18:15:29.062810 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.062814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.063851 (+ 1037us) server_negotiation.cc:407] Connection header received
0810 18:15:29.064667 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.064669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.064718 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.064786 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:29.065917 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.066526 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.067893 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.068142 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.068226 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.070986 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:29.070991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:29.070993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:29.071023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:29.072535 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:29.073010 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:29.073014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:29.073017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:29.073124 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:29.073472 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:29.073475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:29.073477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:29.073708 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:29.073811 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.074277 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.074413 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:15:29.075359 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.063037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:39755 (local address 127.23.65.65:54761)
0810 18:15:29.063695 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:29.063737 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:29.064495 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:29.064923 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:29.064929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:29.065269 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:29.065741 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.065769 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.066642 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.066646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.067777 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.067788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.068296 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.068300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.068486 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.069061 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:29.069082 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:29.070816 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:29.072642 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:29.072647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:29.072659 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:29.072905 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:29.073206 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:29.073208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:29.073209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:29.073380 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:29.073802 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:29.073807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:29.074035 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 18:15:29.074186 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 18:15:29.074443 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":114,"threads_started":1}
I20260810 18:15:29.076565 21225 heartbeater.cc:347] Connected to a master server at 127.23.65.126:39755
I20260810 18:15:29.076797 21225 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:15:29.077258 21225 heartbeater.cc:511] Master 127.23.65.126:39755 requested a full tablet report, sending...
I20260810 18:15:29.078749 20923 ts_manager.cc:194] Registered new tserver with Master: cc53210eb47d4cc9bcc900ccff3969d3 (127.23.65.65:39167)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
I20260810 18:15:29.080031 20923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:54761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:15:29.094592 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:39755
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:29.206329 21234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:29.206585 21234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:29.206646 21234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:29.210894 21234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:29.210963 21234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:29.211042 21234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:15:29.216348 21234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:29.217433 21234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:29.218510 21234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:29.229303 21240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:29.229295 21244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:29.229332 21242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:29.229652 21234 server_base.cc:1034] running on GCE node
I20260810 18:15:29.230086 21234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:29.230755 21234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:29.231915 21234 hybrid_clock.cc:648] HybridClock initialized: now 1786385729231900 us; error 27 us; skew 500 ppm
Aug 10 18:15:29 dist-test-slave-jpj1 krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385729, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:29.234807 21234 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:15:29.235754 21234 webserver.cc:460] Webserver started at http://127.23.65.66:38793/ using document root <none> and password file <none>
I20260810 18:15:29.236292 21234 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:29.236356 21234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:29.236559 21234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:29.238579 21234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "ad9554814613423a94677ad3ea7da3a4"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "19b2c359ebf9dbe7d4f095b294589f41"
server_key_iv: "f3b34fced247bb301878647c4ea178a0"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.239118 21234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "ad9554814613423a94677ad3ea7da3a4"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "19b2c359ebf9dbe7d4f095b294589f41"
server_key_iv: "f3b34fced247bb301878647c4ea178a0"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.242686 21234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 18:15:29.245031 21256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.246188 21234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:15:29.246286 21234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "ad9554814613423a94677ad3ea7da3a4"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "19b2c359ebf9dbe7d4f095b294589f41"
server_key_iv: "f3b34fced247bb301878647c4ea178a0"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.246381 21234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:29.266402 21234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:29.269039 21234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:29.269212 21234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:29.269819 21234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:29.270638 21234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:29.270686 21234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.270730 21234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:29.270766 21234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.279217 21234 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:34815
I20260810 18:15:29.279251 21422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:34815 every 8 connection(s)
I20260810 18:15:29.280089 21234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:15:29.280918 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 21234
I20260810 18:15:29.281060 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:15:29.281409 23813 external_mini_cluster.cc:1428] Setting key 3398e973c1d3f1cdfedabf98be72b56b
Aug 10 18:15:29 dist-test-slave-jpj1 krb5kdc[20852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385729, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
I20260810 18:15:29.294130 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.281967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:57105 (local address 127.23.65.126:39755)
0810 18:15:29.282105 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.282109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.283420 (+ 1311us) server_negotiation.cc:407] Connection header received
0810 18:15:29.284478 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.284481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.284524 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.284588 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:29.285854 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.286613 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.287742 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.288089 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.288198 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.290788 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:29.290791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:29.290793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:29.290817 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:29.292184 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:29.292660 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:29.292662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:29.292663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:29.292776 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:29.293106 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:29.293108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:29.293109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:29.293290 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:29.293361 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.293876 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.293979 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
I20260810 18:15:29.294946 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.282365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:39755 (local address 127.23.65.66:57105)
0810 18:15:29.283257 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:29.283319 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:29.284276 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:29.284767 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:29.284776 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:29.285214 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:29.285688 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.285702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.286728 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.286730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.287630 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.287640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.288233 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.288236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.288373 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.288899 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:29.288915 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:29.290595 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:29.292293 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:29.292298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:29.292309 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:29.292547 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:29.292888 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:29.292891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:29.292893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:29.293036 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:29.293383 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:29.293389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:29.293643 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 18:15:29.293798 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 18:15:29.294049 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":117,"threads_started":1}
I20260810 18:15:29.296125 21424 heartbeater.cc:347] Connected to a master server at 127.23.65.126:39755
I20260810 18:15:29.296357 21424 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:29.296861 21424 heartbeater.cc:511] Master 127.23.65.126:39755 requested a full tablet report, sending...
I20260810 18:15:29.297909 20923 ts_manager.cc:194] Registered new tserver with Master: ad9554814613423a94677ad3ea7da3a4 (127.23.65.66:34815)
I20260810 18:15:29.298481 20923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:57105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:15:29.309764 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:39755
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:29.419680 21429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:29.419929 21429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:29.419973 21429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:29.423928 21429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:29.423980 21429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:29.424046 21429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:15:29.429186 21429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:29.430213 21429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:29.431195 21429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:29.440564 21435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:29.440578 21437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:29.440596 21434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:29.440769 21429 server_base.cc:1034] running on GCE node
I20260810 18:15:29.441258 21429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:29.441862 21429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:29.443015 21429 hybrid_clock.cc:648] HybridClock initialized: now 1786385729442987 us; error 41 us; skew 500 ppm
Aug 10 18:15:29 dist-test-slave-jpj1 krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385729, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:29.445605 21429 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:15:29.446437 21429 webserver.cc:460] Webserver started at http://127.23.65.67:43907/ using document root <none> and password file <none>
I20260810 18:15:29.446940 21429 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:29.446996 21429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:29.447247 21429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:29.449273 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "17159dea7ba546f099d1cb66db48a3ee"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "c3d22fc2b0cb3964d9409c8ae2e1a840"
server_key_iv: "e0406d3d558137bcef5ca7e7b1873818"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.449793 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "17159dea7ba546f099d1cb66db48a3ee"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "c3d22fc2b0cb3964d9409c8ae2e1a840"
server_key_iv: "e0406d3d558137bcef5ca7e7b1873818"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.453334 21429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:15:29.455615 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.456808 21429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:29.456923 21429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "17159dea7ba546f099d1cb66db48a3ee"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "c3d22fc2b0cb3964d9409c8ae2e1a840"
server_key_iv: "e0406d3d558137bcef5ca7e7b1873818"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.457012 21429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:29.473796 21429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:29.476447 21429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:29.476613 21429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:29.477157 21429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:29.477947 21429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:29.477995 21429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.478047 21429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:29.478075 21429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.486176 21429 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:46553
I20260810 18:15:29.486236 21620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:46553 every 8 connection(s)
I20260810 18:15:29.487067 21429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:15:29.495769 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 21429
I20260810 18:15:29.495919 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:15:29.496227 23813 external_mini_cluster.cc:1428] Setting key e9f805e89ae1134ef36ab6a0c8cb826a
Aug 10 18:15:29 dist-test-slave-jpj1 krb5kdc[20852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385729, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:29.500259 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.488856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:59783 (local address 127.23.65.126:39755)
0810 18:15:29.489000 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.489005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.490076 (+ 1071us) server_negotiation.cc:407] Connection header received
0810 18:15:29.490934 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.490938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.490989 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.491066 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:29.492374 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.492890 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.493949 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.494167 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.494253 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.497024 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:29.497028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:29.497030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:29.497058 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:29.498442 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:29.498950 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:29.498953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:29.498954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:29.499028 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:29.499277 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:29.499279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:29.499280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:29.499444 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:29.499515 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.500020 (+ 505us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.500121 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:15:29.501092 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.489274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:39755 (local address 127.23.65.67:59783)
0810 18:15:29.489926 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:29.489968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:29.490705 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:29.491305 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:29.491312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:29.491749 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:29.492254 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.492268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.492978 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.492980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.493845 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.493855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.494340 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.494344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.494490 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.495052 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:29.495069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:29.496812 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:29.498556 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:29.498562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:29.498576 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:29.498868 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:29.499086 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:29.499088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:29.499089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:29.499211 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:29.499516 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:29.499521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:29.499800 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 18:15:29.499969 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 18:15:29.500242 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":120,"threads_started":1}
I20260810 18:15:29.502118 21621 heartbeater.cc:347] Connected to a master server at 127.23.65.126:39755
I20260810 18:15:29.502362 21621 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:29.502883 21621 heartbeater.cc:511] Master 127.23.65.126:39755 requested a full tablet report, sending...
I20260810 18:15:29.503871 20923 ts_manager.cc:194] Registered new tserver with Master: 17159dea7ba546f099d1cb66db48a3ee (127.23.65.67:46553)
I20260810 18:15:29.504402 20923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:59783
I20260810 18:15:29.510280 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:15:29.518676 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.512263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.23.65.126:39755)
0810 18:15:29.512366 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.512370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.512540 (+ 170us) server_negotiation.cc:407] Connection header received
0810 18:15:29.512649 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.512652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.512687 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.512741 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:29.513368 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.513840 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.514574 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.514769 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.514829 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.516217 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:29.516220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:29.516222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:29.516241 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:29.517319 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:29.517732 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:29.517734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:29.517736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:29.517799 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:29.518024 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:29.518025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:29.518026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:29.518199 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:29.518315 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.518492 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.518564 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:15:29.521332 20923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:15:29.524261 20923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:15:29.538600 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.532461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:46553 (local address 127.0.0.1:35742)
0810 18:15:29.534188 (+ 1727us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:29.534202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:29.534284 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:29.534978 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:29.534981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:29.535000 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:29.535086 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.535091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.536133 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.536136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.537230 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.537238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.538084 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.538087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.538202 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.538241 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 18:15:29.538339 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 18:15:29.538452 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1663,"mutex_wait_us":88,"thread_start_us":84,"threads_started":1}
I20260810 18:15:29.538853 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.532343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:34815 (local address 127.0.0.1:41944)
0810 18:15:29.532901 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:29.532953 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:29.533099 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:29.534141 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:29.534148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:29.534174 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:29.534317 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.534327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.535728 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.535732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.537236 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.537245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.538486 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.538488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.538553 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.538570 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 18:15:29.538631 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 18:15:29.538708 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":97,"thread_start_us":73,"threads_started":1}
I20260810 18:15:29.539121 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.533018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.23.65.66:34815)
0810 18:15:29.533401 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.533414 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.533440 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:15:29.533521 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.533527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.533740 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.533922 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:29.534460 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.535589 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.537359 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.538187 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.538289 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.538432 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.538686 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.538782 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":129,"threads_started":1}
I20260810 18:15:29.539561 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.533586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35742 (local address 127.23.65.67:46553)
0810 18:15:29.534514 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.534527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.534549 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:15:29.534623 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.534629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.534835 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.534987 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:29.535178 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.536026 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.537375 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.537969 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.538068 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.538219 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.539300 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.539401 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":141,"threads_started":1}
I20260810 18:15:29.540519 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.533926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:39167 (local address 127.0.0.1:58784)
0810 18:15:29.534853 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:29.534866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:29.534960 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:29.535611 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:29.535614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:29.535634 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:29.535722 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.535728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.537154 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.537158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.538253 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.538263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.539461 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.539465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.539586 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.539611 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:15:29.539682 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 18:15:29.540404 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":103,"threads_started":1}
I20260810 18:15:29.541013 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.534358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.23.65.65:39167)
0810 18:15:29.534800 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.534815 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.535206 (+ 391us) server_negotiation.cc:407] Connection header received
0810 18:15:29.535282 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.535287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.535461 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.535632 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:29.535829 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.536995 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.538364 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.539153 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.539270 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.539398 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.540787 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.540884 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":91,"threads_started":1}
I20260810 18:15:29.542199 21325 tablet_service.cc:1467] Processing CreateTablet for tablet 0c991e529427442daf79a9bbba30a6b1 (DEFAULT_TABLE table=test-table [id=882e1cd888044822b1900bb4bd3d9a84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:29.542199 21520 tablet_service.cc:1467] Processing CreateTablet for tablet 0c991e529427442daf79a9bbba30a6b1 (DEFAULT_TABLE table=test-table [id=882e1cd888044822b1900bb4bd3d9a84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:29.543370 21325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c991e529427442daf79a9bbba30a6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:29.543375 21520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c991e529427442daf79a9bbba30a6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:29.543217 21128 tablet_service.cc:1467] Processing CreateTablet for tablet 0c991e529427442daf79a9bbba30a6b1 (DEFAULT_TABLE table=test-table [id=882e1cd888044822b1900bb4bd3d9a84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:29.544178 21128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c991e529427442daf79a9bbba30a6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:29.549954 21645 tablet_bootstrap.cc:492] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3: Bootstrap starting.
I20260810 18:15:29.551380 21646 tablet_bootstrap.cc:492] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: Bootstrap starting.
I20260810 18:15:29.551851 21647 tablet_bootstrap.cc:492] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: Bootstrap starting.
I20260810 18:15:29.552263 21645 tablet_bootstrap.cc:654] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:29.553486 21646 tablet_bootstrap.cc:654] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:29.553582 21645 log.cc:824] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:29.553849 21647 tablet_bootstrap.cc:654] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:29.554750 21646 log.cc:824] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:29.554991 21647 log.cc:824] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:29.555452 21645 tablet_bootstrap.cc:492] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3: No bootstrap required, opened a new log
I20260810 18:15:29.555675 21645 ts_tablet_manager.cc:1397] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:15:29.556845 21646 tablet_bootstrap.cc:492] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: No bootstrap required, opened a new log
I20260810 18:15:29.556915 21647 tablet_bootstrap.cc:492] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: No bootstrap required, opened a new log
I20260810 18:15:29.557011 21646 ts_tablet_manager.cc:1397] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:15:29.557050 21647 ts_tablet_manager.cc:1397] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:15:29.559836 21645 raft_consensus.cc:348] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } }
I20260810 18:15:29.560175 21645 raft_consensus.cc:374] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:29.560245 21645 raft_consensus.cc:729] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc53210eb47d4cc9bcc900ccff3969d3, State: Initialized, Role: FOLLOWER
I20260810 18:15:29.560070 21647 raft_consensus.cc:348] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } }
I20260810 18:15:29.560310 21647 raft_consensus.cc:374] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:29.560384 21647 raft_consensus.cc:729] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17159dea7ba546f099d1cb66db48a3ee, State: Initialized, Role: FOLLOWER
I20260810 18:15:29.560629 21646 raft_consensus.cc:348] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } }
I20260810 18:15:29.560722 21645 consensus_queue.cc:260] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } }
I20260810 18:15:29.560770 21647 consensus_queue.cc:260] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } }
I20260810 18:15:29.560889 21646 raft_consensus.cc:374] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:29.560945 21646 raft_consensus.cc:729] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9554814613423a94677ad3ea7da3a4, State: Initialized, Role: FOLLOWER
I20260810 18:15:29.561362 21646 consensus_queue.cc:260] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } }
I20260810 18:15:29.561499 21621 heartbeater.cc:503] Master 127.23.65.126:39755 was elected leader, sending a full tablet report...
I20260810 18:15:29.561800 21645 ts_tablet_manager.cc:1428] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260810 18:15:29.561955 21226 tablet.cc:2369] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:15:29.562219 21646 ts_tablet_manager.cc:1428] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:15:29.562512 21424 heartbeater.cc:503] Master 127.23.65.126:39755 was elected leader, sending a full tablet report...
I20260810 18:15:29.562654 21225 heartbeater.cc:503] Master 127.23.65.126:39755 was elected leader, sending a full tablet report...
I20260810 18:15:29.563680 21647 ts_tablet_manager.cc:1428] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:15:29.600627 21663 raft_consensus.cc:482] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:29.600854 21663 raft_consensus.cc:504] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } }
I20260810 18:15:29.602089 21663 leader_election.cc:288] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17159dea7ba546f099d1cb66db48a3ee (127.23.65.67:46553), cc53210eb47d4cc9bcc900ccff3969d3 (127.23.65.65:39167)
I20260810 18:15:29.606659 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.602447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:35363 (local address 127.23.65.67:46553)
0810 18:15:29.602620 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.602626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.602642 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:15:29.602698 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.602701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.602745 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.602810 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:29.603138 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.603915 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.605429 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.606029 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.606148 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.606200 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.606463 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.606530 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:15:29.606659 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.602430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:51961 (local address 127.23.65.65:39167)
0810 18:15:29.602567 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.602571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.602919 (+ 348us) server_negotiation.cc:407] Connection header received
0810 18:15:29.603019 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.603021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.603062 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.603134 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:29.603362 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.604094 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.605427 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.606029 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.606148 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.606200 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.606464 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.606529 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:15:29.606711 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.602299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:46553 (local address 127.23.65.66:35363)
0810 18:15:29.602428 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:29.602444 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:29.602536 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:29.602856 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:29.602860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:29.602881 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:29.603031 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.603038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.604021 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.604023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.605296 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.605308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.606137 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.606141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.606329 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.606366 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 18:15:29.606441 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:15:29.606559 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":123,"spinlock_wait_cycles":14976}
I20260810 18:15:29.606724 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.602304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:39167 (local address 127.23.65.66:51961)
0810 18:15:29.602802 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:29.602827 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:29.602935 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:29.603153 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:29.603155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:29.603171 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:29.603265 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.603269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.604181 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.604184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.605327 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:29.605334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.606137 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.606141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:29.606274 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.606372 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 18:15:29.606442 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:15:29.606555 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":194,"spinlock_wait_cycles":384,"thread_start_us":113,"threads_started":1}
I20260810 18:15:29.607429 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c991e529427442daf79a9bbba30a6b1" candidate_uuid: "ad9554814613423a94677ad3ea7da3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17159dea7ba546f099d1cb66db48a3ee" is_pre_election: true
I20260810 18:15:29.607566 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c991e529427442daf79a9bbba30a6b1" candidate_uuid: "ad9554814613423a94677ad3ea7da3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" is_pre_election: true
I20260810 18:15:29.607743 21550 raft_consensus.cc:2454] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad9554814613423a94677ad3ea7da3a4 in term 0.
I20260810 18:15:29.607851 21154 raft_consensus.cc:2454] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad9554814613423a94677ad3ea7da3a4 in term 0.
I20260810 18:15:29.608245 21259 leader_election.cc:302] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17159dea7ba546f099d1cb66db48a3ee, ad9554814613423a94677ad3ea7da3a4, cc53210eb47d4cc9bcc900ccff3969d3; no voters:
I20260810 18:15:29.608527 21663 raft_consensus.cc:2781] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:15:29.608613 21663 raft_consensus.cc:482] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:15:29.608655 21663 raft_consensus.cc:3037] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:29.609666 21663 raft_consensus.cc:504] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } }
I20260810 18:15:29.609979 21663 leader_election.cc:288] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [CANDIDATE]: Term 1 election: Requested vote from peers 17159dea7ba546f099d1cb66db48a3ee (127.23.65.67:46553), cc53210eb47d4cc9bcc900ccff3969d3 (127.23.65.65:39167)
I20260810 18:15:29.610301 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c991e529427442daf79a9bbba30a6b1" candidate_uuid: "ad9554814613423a94677ad3ea7da3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17159dea7ba546f099d1cb66db48a3ee"
I20260810 18:15:29.610361 21154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c991e529427442daf79a9bbba30a6b1" candidate_uuid: "ad9554814613423a94677ad3ea7da3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc53210eb47d4cc9bcc900ccff3969d3"
I20260810 18:15:29.610442 21550 raft_consensus.cc:3037] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:29.610466 21154 raft_consensus.cc:3037] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:29.611743 21550 raft_consensus.cc:2454] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad9554814613423a94677ad3ea7da3a4 in term 1.
I20260810 18:15:29.611743 21154 raft_consensus.cc:2454] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad9554814613423a94677ad3ea7da3a4 in term 1.
I20260810 18:15:29.612092 21259 leader_election.cc:302] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17159dea7ba546f099d1cb66db48a3ee, ad9554814613423a94677ad3ea7da3a4, cc53210eb47d4cc9bcc900ccff3969d3; no voters:
I20260810 18:15:29.612250 21663 raft_consensus.cc:2781] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:29.612475 21663 raft_consensus.cc:686] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 1 LEADER]: Becoming Leader. State: Replica: ad9554814613423a94677ad3ea7da3a4, State: Running, Role: LEADER
I20260810 18:15:29.612805 21663 consensus_queue.cc:237] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } }
I20260810 18:15:29.615737 20921 catalog_manager.cc:5495] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad9554814613423a94677ad3ea7da3a4 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "ad9554814613423a94677ad3ea7da3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad9554814613423a94677ad3ea7da3a4" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 34815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 } health_report { overall_health: UNKNOWN } } }
I20260810 18:15:29.639310 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.635904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.23.65.66:34815)
0810 18:15:29.636014 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.636018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.636032 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:15:29.636186 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.636189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.636243 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.636342 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:15:29.636572 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.637062 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.637904 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.638118 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.638206 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.638504 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:15:29.638895 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:15:29.639028 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.639149 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.639188 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 18:15:29.644798 21154 raft_consensus.cc:1261] T 0c991e529427442daf79a9bbba30a6b1 P cc53210eb47d4cc9bcc900ccff3969d3 [term 1 FOLLOWER]: Refusing update from remote peer ad9554814613423a94677ad3ea7da3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:29.644805 21550 raft_consensus.cc:1261] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [term 1 FOLLOWER]: Refusing update from remote peer ad9554814613423a94677ad3ea7da3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:29.645419 21666 consensus_queue.cc:1035] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17159dea7ba546f099d1cb66db48a3ee" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:29.645597 21663 consensus_queue.cc:1035] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc53210eb47d4cc9bcc900ccff3969d3" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 39167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:29.662240 20921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:29.662453 20921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:29.664790 20921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa5e6d165bc441fe9ff454d47dbe45f4: Sending DeleteTablet for 3 replicas of tablet 0c991e529427442daf79a9bbba30a6b1
I20260810 18:15:29.665525 21325 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c991e529427442daf79a9bbba30a6b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:29 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:41944
I20260810 18:15:29.665581 21520 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c991e529427442daf79a9bbba30a6b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:29 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:35742
I20260810 18:15:29.665658 21128 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c991e529427442daf79a9bbba30a6b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:29 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:58784
I20260810 18:15:29.666160 21686 tablet_replica.cc:330] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: stopping tablet replica
I20260810 18:15:29.666213 21688 tablet_replica.cc:330] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: stopping tablet replica
I20260810 18:15:29.666538 21688 raft_consensus.cc:2229] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:15:29.666538 21686 raft_consensus.cc:2229] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:15:29.666843 21688 raft_consensus.cc:2258] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:29.666874 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 21021
I20260810 18:15:29.667088 21686 raft_consensus.cc:2258] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:29.668576 21686 ts_tablet_manager.cc:1905] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:29.671139 21686 ts_tablet_manager.cc:1918] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:29.671281 21686 log.cc:1196] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/0c991e529427442daf79a9bbba30a6b1
I20260810 18:15:29.671646 21686 ts_tablet_manager.cc:1939] T 0c991e529427442daf79a9bbba30a6b1 P ad9554814613423a94677ad3ea7da3a4: Deleting consensus metadata
I20260810 18:15:29.673194 20903 catalog_manager.cc:4841] TS ad9554814613423a94677ad3ea7da3a4 (127.23.65.66:34815): tablet 0c991e529427442daf79a9bbba30a6b1 (table test-table [id=882e1cd888044822b1900bb4bd3d9a84]) successfully deleted
I20260810 18:15:29.674923 21688 ts_tablet_manager.cc:1905] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:29.677793 21688 ts_tablet_manager.cc:1918] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:29.677902 21688 log.cc:1196] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/0c991e529427442daf79a9bbba30a6b1
I20260810 18:15:29.678295 21688 ts_tablet_manager.cc:1939] T 0c991e529427442daf79a9bbba30a6b1 P 17159dea7ba546f099d1cb66db48a3ee: Deleting consensus metadata
I20260810 18:15:29.679603 20906 catalog_manager.cc:4841] TS 17159dea7ba546f099d1cb66db48a3ee (127.23.65.67:46553): tablet 0c991e529427442daf79a9bbba30a6b1 (table test-table [id=882e1cd888044822b1900bb4bd3d9a84]) successfully deleted
W20260810 18:15:29.679687 20907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:39167 (error 108)
I20260810 18:15:29.680532 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 21234
I20260810 18:15:29.680752 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.680445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:39167 (local address 127.0.0.1:58796)
0810 18:15:29.680591 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:29.680666 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:39167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 18:15:29.684712 20907 catalog_manager.cc:4584] TS cc53210eb47d4cc9bcc900ccff3969d3 (127.23.65.65:39167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c991e529427442daf79a9bbba30a6b1: Network error: Client connection negotiation failed: client connection to 127.23.65.65:39167: connect: Connection refused (error 111)
I20260810 18:15:29.694388 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 21429
I20260810 18:15:29.707753 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 20871
2026-08-10T18:15:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2382 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T18:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:29Z Disabled control of system clock
I20260810 18:15:29.741264 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40797
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:40797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:15:29.855871 21699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:29.856148 21699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:29.856189 21699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:29.860077 21699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:29.860126 21699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:29.860140 21699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:29.860170 21699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:29.860196 21699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:29.864794 21699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:40797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:29.865760 21699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:29.867106 21699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:29.875088 21711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:29.875093 21709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:29.875095 21708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:29.875422 21699 server_base.cc:1034] running on GCE node
I20260810 18:15:29.876240 21699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:29.877295 21699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:29.878479 21699 hybrid_clock.cc:648] HybridClock initialized: now 1786385729878451 us; error 38 us; skew 500 ppm
I20260810 18:15:29.880331 21699 webserver.cc:460] Webserver started at http://127.23.65.126:40139/ using document root <none> and password file <none>
I20260810 18:15:29.880877 21699 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:29.880944 21699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:29.881134 21699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:29.883133 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "236dbaef528b4d4eb607841103043c59"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "3ffcfb4daa6ab235c52550ce846215f2"
server_key_iv: "5160940fd14903327436c7f2e1b84dd2"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.883700 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "236dbaef528b4d4eb607841103043c59"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "3ffcfb4daa6ab235c52550ce846215f2"
server_key_iv: "5160940fd14903327436c7f2e1b84dd2"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.887516 21699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:15:29.889832 21721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:29.890972 21699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:29.891088 21699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "236dbaef528b4d4eb607841103043c59"
format_stamp: "Formatted at 2026-08-10 18:15:29 on dist-test-slave-jpj1"
server_key: "3ffcfb4daa6ab235c52550ce846215f2"
server_key_iv: "5160940fd14903327436c7f2e1b84dd2"
server_key_version: "encryptionkey@0"
I20260810 18:15:29.891181 21699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:29.905287 21699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:29.905890 21699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:29.906049 21699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:29.912799 21699 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:40797
I20260810 18:15:29.912815 21804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:40797 every 8 connection(s)
I20260810 18:15:29.913730 21699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:29.916766 21806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:29.916890 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 21699
I20260810 18:15:29.917001 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:29.917279 23813 external_mini_cluster.cc:1428] Setting key 15d6d1678040981fef0f7ae4ae483fd8
I20260810 18:15:29.923923 21806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59: Bootstrap starting.
I20260810 18:15:29.926469 21806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:29.927357 21806 log.cc:824] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:29.927511 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:29.918618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.23.65.126:40797)
0810 18:15:29.919300 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:29.919331 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:29.919376 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:15:29.920116 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:29.920140 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:29.920669 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:29.921039 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:29.921484 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.922937 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.923847 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:29.924504 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:29.924659 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:29.925495 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:29.925501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:29.925515 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:29.925991 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:29.926062 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:29.926089 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:29.926177 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:29.926295 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 18:15:29.926576 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":133,"threads_started":1}
I20260810 18:15:29.929487 21806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59: No bootstrap required, opened a new log
I20260810 18:15:29.933199 21806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236dbaef528b4d4eb607841103043c59" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40797 } }
I20260810 18:15:29.933414 21806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:29.933454 21806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 236dbaef528b4d4eb607841103043c59, State: Initialized, Role: FOLLOWER
I20260810 18:15:29.933828 21806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236dbaef528b4d4eb607841103043c59" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40797 } }
I20260810 18:15:29.933950 21806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:29.934010 21806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:29.934089 21806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:29.934885 21806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236dbaef528b4d4eb607841103043c59" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40797 } }
I20260810 18:15:29.935197 21806 leader_election.cc:302] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 236dbaef528b4d4eb607841103043c59; no voters:
I20260810 18:15:29.935571 21806 leader_election.cc:288] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:29.935663 21816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:29.935910 21816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [term 1 LEADER]: Becoming Leader. State: Replica: 236dbaef528b4d4eb607841103043c59, State: Running, Role: LEADER
I20260810 18:15:29.936204 21816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236dbaef528b4d4eb607841103043c59" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40797 } }
I20260810 18:15:29.936743 21806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:29.937714 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 236dbaef528b4d4eb607841103043c59. Latest consensus state: current_term: 1 leader_uuid: "236dbaef528b4d4eb607841103043c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236dbaef528b4d4eb607841103043c59" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40797 } } }
I20260810 18:15:29.937700 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "236dbaef528b4d4eb607841103043c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236dbaef528b4d4eb607841103043c59" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40797 } } }
I20260810 18:15:29.937848 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:29.937848 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:29.938197 21829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:29.941251 21829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:29.946430 21829 catalog_manager.cc:1347] Generated new cluster ID: c3729386ffe94c5cae0c0bf4b6815dd0
I20260810 18:15:29.946483 21829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:29.978818 21829 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:29.979882 21829 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:29.985867 21829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59: Generated new TSK 0
I20260810 18:15:29.986546 21829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:29.998353 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40797
--builtin_ntp_servers=127.23.65.84:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:15:30.112468 21842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:30.112718 21842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:30.112762 21842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:30.116743 21842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:30.116796 21842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:30.116863 21842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:15:30.121511 21842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:30.122529 21842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:30.123932 21842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:30.133129 21853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:30.133206 21852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:30.133301 21855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:30.133458 21842 server_base.cc:1034] running on GCE node
I20260810 18:15:30.133860 21842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:30.134516 21842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:30.135672 21842 hybrid_clock.cc:648] HybridClock initialized: now 1786385730135648 us; error 37 us; skew 500 ppm
I20260810 18:15:30.137516 21842 webserver.cc:460] Webserver started at http://127.23.65.65:35023/ using document root <none> and password file <none>
I20260810 18:15:30.138121 21842 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:30.138188 21842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:30.138386 21842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:30.140506 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "177dc272268040a0ad9e31b89dd35bed"
format_stamp: "Formatted at 2026-08-10 18:15:30 on dist-test-slave-jpj1"
server_key: "cf4e33186c4c6c8e751269d6386eba4e"
server_key_iv: "7d979195696a3bbca4b2f7006b35e75b"
server_key_version: "encryptionkey@0"
I20260810 18:15:30.141037 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "177dc272268040a0ad9e31b89dd35bed"
format_stamp: "Formatted at 2026-08-10 18:15:30 on dist-test-slave-jpj1"
server_key: "cf4e33186c4c6c8e751269d6386eba4e"
server_key_iv: "7d979195696a3bbca4b2f7006b35e75b"
server_key_version: "encryptionkey@0"
I20260810 18:15:30.144515 21842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:30.146863 21863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:30.148079 21842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:30.148178 21842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "177dc272268040a0ad9e31b89dd35bed"
format_stamp: "Formatted at 2026-08-10 18:15:30 on dist-test-slave-jpj1"
server_key: "cf4e33186c4c6c8e751269d6386eba4e"
server_key_iv: "7d979195696a3bbca4b2f7006b35e75b"
server_key_version: "encryptionkey@0"
I20260810 18:15:30.148276 21842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:30.167908 21842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:30.168619 21842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:30.168830 21842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:30.169443 21842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:30.170408 21842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:30.170465 21842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:30.170521 21842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:30.170549 21842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:30.179445 21842 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:37557
I20260810 18:15:30.179610 22044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:37557 every 8 connection(s)
I20260810 18:15:30.180408 21842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:15:30.185534 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 21842
I20260810 18:15:30.185667 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:15:30.185971 23813 external_mini_cluster.cc:1428] Setting key e5641932466646a45f3843fc12449064
I20260810 18:15:30.188059 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40797
--builtin_ntp_servers=127.23.65.84:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:15:30.189841 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.182180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:60139 (local address 127.23.65.126:40797)
0810 18:15:30.182327 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.182332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.183426 (+ 1094us) server_negotiation.cc:407] Connection header received
0810 18:15:30.184800 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.184802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.184852 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.184925 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:30.185877 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.186552 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.187916 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.188150 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.188253 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.189363 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:30.189366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:30.189369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:30.189382 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:30.189404 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:30.189412 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:30.189469 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.189617 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.189684 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:15:30.190702 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.182613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40797 (local address 127.23.65.65:60139)
0810 18:15:30.183277 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:30.183320 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:30.184611 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:30.185076 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:30.185083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:30.185165 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:30.185712 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.185732 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.186696 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.186699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.187800 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.187812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.188401 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.188404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.188549 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.189117 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:15:30.189119 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:30.189157 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:15:30.189219 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:30.189479 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:30.189489 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:30.189495 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:15:30.189608 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 18:15:30.189912 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260810 18:15:30.191903 22045 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40797
I20260810 18:15:30.192124 22045 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:30.192605 22045 heartbeater.cc:511] Master 127.23.65.126:40797 requested a full tablet report, sending...
I20260810 18:15:30.194046 21747 ts_manager.cc:194] Registered new tserver with Master: 177dc272268040a0ad9e31b89dd35bed (127.23.65.65:37557)
I20260810 18:15:30.195195 21747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.65:60139
W20260810 18:15:30.301981 22051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:30.302271 22051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:30.302326 22051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:30.306366 22051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:30.306416 22051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:30.306491 22051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:15:30.311156 22051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:30.312192 22051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:30.313669 22051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:30.322945 22060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:30.322942 22059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:30.322945 22062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:30.323335 22051 server_base.cc:1034] running on GCE node
I20260810 18:15:30.323730 22051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:30.324297 22051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:30.325470 22051 hybrid_clock.cc:648] HybridClock initialized: now 1786385730325448 us; error 35 us; skew 500 ppm
I20260810 18:15:30.331351 22051 webserver.cc:460] Webserver started at http://127.23.65.66:36591/ using document root <none> and password file <none>
I20260810 18:15:30.331947 22051 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:30.332011 22051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:30.332185 22051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:30.334195 22051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "07fe34f5401946dbbe5c2b5b86f73623"
format_stamp: "Formatted at 2026-08-10 18:15:30 on dist-test-slave-jpj1"
server_key: "c578b5f379db13312855bf5c64671280"
server_key_iv: "2fa71e5ab16ca8b00f55e24244fcc0f3"
server_key_version: "encryptionkey@0"
I20260810 18:15:30.334821 22051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "07fe34f5401946dbbe5c2b5b86f73623"
format_stamp: "Formatted at 2026-08-10 18:15:30 on dist-test-slave-jpj1"
server_key: "c578b5f379db13312855bf5c64671280"
server_key_iv: "2fa71e5ab16ca8b00f55e24244fcc0f3"
server_key_version: "encryptionkey@0"
I20260810 18:15:30.339967 22051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 18:15:30.344832 22070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:30.346297 22051 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260810 18:15:30.346421 22051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "07fe34f5401946dbbe5c2b5b86f73623"
format_stamp: "Formatted at 2026-08-10 18:15:30 on dist-test-slave-jpj1"
server_key: "c578b5f379db13312855bf5c64671280"
server_key_iv: "2fa71e5ab16ca8b00f55e24244fcc0f3"
server_key_version: "encryptionkey@0"
I20260810 18:15:30.346517 22051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:30.374135 22051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:30.375114 22051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:30.375329 22051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:30.376150 22051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:30.377471 22051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:30.377542 22051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:30.377610 22051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:30.377641 22051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:30.389272 22268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:41253 every 8 connection(s)
I20260810 18:15:30.389252 22051 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:41253
I20260810 18:15:30.390728 22051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:15:30.397198 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 22051
I20260810 18:15:30.397358 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:15:30.397708 23813 external_mini_cluster.cc:1428] Setting key ef529fd953f1391b027f95764e4d38aa
I20260810 18:15:30.399961 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40797
--builtin_ntp_servers=127.23.65.84:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:15:30.400709 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.392748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:54713 (local address 127.23.65.126:40797)
0810 18:15:30.392895 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.392899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.393916 (+ 1017us) server_negotiation.cc:407] Connection header received
0810 18:15:30.395446 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.395450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.395505 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.395593 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:30.396604 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.397143 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.398646 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.398964 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.399063 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.400286 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:30.400303 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:30.400305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:30.400317 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:30.400338 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:30.400346 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:30.400401 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.400545 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.400595 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260810 18:15:30.401886 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.393121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40797 (local address 127.23.65.66:54713)
0810 18:15:30.393764 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:30.393807 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:30.395205 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:30.395881 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:30.395888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:30.395974 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:30.396475 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.396488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.397263 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.397267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.398531 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.398541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.399114 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.399119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.399289 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.399994 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:15:30.399997 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:30.400055 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:15:30.400144 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:30.400418 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:30.400426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:30.400432 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:15:30.400552 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 18:15:30.400866 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260810 18:15:30.402974 22269 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40797
I20260810 18:15:30.403231 22269 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:30.403811 22269 heartbeater.cc:511] Master 127.23.65.126:40797 requested a full tablet report, sending...
I20260810 18:15:30.404800 21747 ts_manager.cc:194] Registered new tserver with Master: 07fe34f5401946dbbe5c2b5b86f73623 (127.23.65.66:41253)
I20260810 18:15:30.405354 21747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.66:54713
W20260810 18:15:30.513469 22272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:30.513723 22272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:30.513775 22272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:30.517714 22272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:30.517766 22272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:30.517836 22272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:15:30.522483 22272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:30.523510 22272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:30.524861 22272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:30.534185 22285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:30.534180 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:30.534176 22280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:30.534776 22272 server_base.cc:1034] running on GCE node
I20260810 18:15:30.535238 22272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:30.536001 22272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:30.537182 22272 hybrid_clock.cc:648] HybridClock initialized: now 1786385730537101 us; error 98 us; skew 500 ppm
I20260810 18:15:30.539000 22272 webserver.cc:460] Webserver started at http://127.23.65.67:34529/ using document root <none> and password file <none>
I20260810 18:15:30.539610 22272 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:30.539680 22272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:30.539884 22272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:30.541906 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "ed7658e306dd460395e7efe9e4d4a324"
format_stamp: "Formatted at 2026-08-10 18:15:30 on dist-test-slave-jpj1"
server_key: "cc2edc9b2827cbe6bd22ba7cff697f94"
server_key_iv: "0390371d58754db8994b620c429b1e7d"
server_key_version: "encryptionkey@0"
I20260810 18:15:30.542451 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "ed7658e306dd460395e7efe9e4d4a324"
format_stamp: "Formatted at 2026-08-10 18:15:30 on dist-test-slave-jpj1"
server_key: "cc2edc9b2827cbe6bd22ba7cff697f94"
server_key_iv: "0390371d58754db8994b620c429b1e7d"
server_key_version: "encryptionkey@0"
I20260810 18:15:30.546025 22272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:15:30.548391 22291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:30.549587 22272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:30.549707 22272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "ed7658e306dd460395e7efe9e4d4a324"
format_stamp: "Formatted at 2026-08-10 18:15:30 on dist-test-slave-jpj1"
server_key: "cc2edc9b2827cbe6bd22ba7cff697f94"
server_key_iv: "0390371d58754db8994b620c429b1e7d"
server_key_version: "encryptionkey@0"
I20260810 18:15:30.549806 22272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:30.564708 22272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:30.565393 22272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:30.565572 22272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:30.566155 22272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:30.567118 22272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:30.567167 22272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:30.567221 22272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:30.567250 22272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:30.575803 22272 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:46587
I20260810 18:15:30.575842 22456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:46587 every 8 connection(s)
I20260810 18:15:30.576699 22272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:15:30.577466 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 22272
I20260810 18:15:30.577594 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:15:30.577867 23813 external_mini_cluster.cc:1428] Setting key e604f6b1020de1cc97089056d54355be
I20260810 18:15:30.585542 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.578492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:47451 (local address 127.23.65.126:40797)
0810 18:15:30.578615 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.578619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.579920 (+ 1301us) server_negotiation.cc:407] Connection header received
0810 18:15:30.581405 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.581407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.581453 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.581517 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:30.582332 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.582858 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.583948 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.584166 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.584242 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.585172 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:30.585175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:30.585177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:30.585188 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:30.585208 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:30.585216 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:30.585268 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.585380 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.585430 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:15:30.586437 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.579004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40797 (local address 127.23.65.67:47451)
0810 18:15:30.579767 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:30.579820 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:30.581212 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:30.581651 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:30.581657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:30.581740 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:30.582229 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.582242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.582947 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.582949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.583834 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.583844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.584275 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.584278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.584423 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.584943 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:15:30.584944 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:30.584984 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:15:30.585044 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:30.585275 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:30.585280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:30.585284 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:15:30.585374 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:15:30.585589 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":109,"threads_started":1}
I20260810 18:15:30.587462 22457 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40797
I20260810 18:15:30.587687 22457 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:30.588158 22457 heartbeater.cc:511] Master 127.23.65.126:40797 requested a full tablet report, sending...
I20260810 18:15:30.589083 21747 ts_manager.cc:194] Registered new tserver with Master: ed7658e306dd460395e7efe9e4d4a324 (127.23.65.67:46587)
I20260810 18:15:30.589605 21747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.67:47451
I20260810 18:15:30.591568 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:15:30.596886 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.593472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.23.65.126:40797)
0810 18:15:30.593594 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.593598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.593738 (+ 140us) server_negotiation.cc:407] Connection header received
0810 18:15:30.593826 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.593828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.593864 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.593923 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:30.594121 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.594619 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.595404 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.595642 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.595716 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.596439 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:30.596442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:30.596444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:30.596455 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:30.596474 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:30.596484 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:30.596591 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.596703 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.596733 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:15:30.599527 21747 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:15:30.602464 21747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:15:30.616185 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.610817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:41253 (local address 127.0.0.1:39124)
0810 18:15:30.611208 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:30.611221 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:30.611355 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:30.612245 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:30.612251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:30.612288 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:30.612442 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.612450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.613473 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.613476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.614597 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.614607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.615628 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.615631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.615741 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.615780 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 18:15:30.615904 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 18:15:30.616024 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"spinlock_wait_cycles":6016,"thread_start_us":83,"threads_started":1}
I20260810 18:15:30.616226 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.611230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.23.65.66:41253)
0810 18:15:30.611800 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.611811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.611838 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:15:30.611917 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.611922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.612087 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.612224 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:30.612578 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.613355 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.614716 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.615238 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.615341 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.615573 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.615949 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.616055 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":91,"threads_started":1}
I20260810 18:15:30.616698 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.611122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:46587 (local address 127.0.0.1:58446)
0810 18:15:30.611422 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:30.611436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:30.611562 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:30.612473 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:30.612476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:30.612493 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:30.612565 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.612570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.613927 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.613930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.615298 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.615305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.616296 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.616300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.616399 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.616425 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:15:30.616475 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 18:15:30.616569 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":20,"thread_start_us":82,"threads_started":1}
I20260810 18:15:30.616911 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.610407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37557 (local address 127.0.0.1:41508)
0810 18:15:30.610776 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:30.610842 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:30.611514 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:30.612244 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:30.612250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:30.612286 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:30.612466 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.612474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.613924 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.613928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.615196 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.615207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.616472 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.616476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.616608 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.616632 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:15:30.616695 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 18:15:30.616777 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":133,"spinlock_wait_cycles":4352,"thread_start_us":94,"threads_started":1}
I20260810 18:15:30.617336 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.610520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.23.65.65:37557)
0810 18:15:30.611007 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.611019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.611048 (+ 29us) server_negotiation.cc:407] Connection header received
0810 18:15:30.611707 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.611715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.611915 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.612049 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:30.612597 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.613792 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.615345 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.616034 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.616133 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.616702 (+ 569us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.616923 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.617163 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":134,"threads_started":1}
I20260810 18:15:30.617547 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.611719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.23.65.67:46587)
0810 18:15:30.612055 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.612065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.612088 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:15:30.612159 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.612163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.612335 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.612487 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:30.612651 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.613797 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.615402 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.616158 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.616969 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.617242 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.617293 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.617393 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
I20260810 18:15:30.619341 22166 tablet_service.cc:1467] Processing CreateTablet for tablet 2813cd0760ce44ffac5752d088aca3ac (DEFAULT_TABLE table=test-table [id=2abb82d51bec4524b82c244210627220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:30.620400 22166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2813cd0760ce44ffac5752d088aca3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:30.620218 21942 tablet_service.cc:1467] Processing CreateTablet for tablet 2813cd0760ce44ffac5752d088aca3ac (DEFAULT_TABLE table=test-table [id=2abb82d51bec4524b82c244210627220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:30.620215 22364 tablet_service.cc:1467] Processing CreateTablet for tablet 2813cd0760ce44ffac5752d088aca3ac (DEFAULT_TABLE table=test-table [id=2abb82d51bec4524b82c244210627220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:30.621284 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2813cd0760ce44ffac5752d088aca3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:30.621284 21942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2813cd0760ce44ffac5752d088aca3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:30.626809 22481 tablet_bootstrap.cc:492] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: Bootstrap starting.
I20260810 18:15:30.628849 22482 tablet_bootstrap.cc:492] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed: Bootstrap starting.
I20260810 18:15:30.630190 22481 tablet_bootstrap.cc:654] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:30.630555 22483 tablet_bootstrap.cc:492] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: Bootstrap starting.
I20260810 18:15:30.631140 22482 tablet_bootstrap.cc:654] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:30.631089 22481 log.cc:824] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:30.632009 22482 log.cc:824] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:30.632658 22483 tablet_bootstrap.cc:654] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:30.633345 22481 tablet_bootstrap.cc:492] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: No bootstrap required, opened a new log
I20260810 18:15:30.633453 22483 log.cc:824] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:30.633528 22481 ts_tablet_manager.cc:1397] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 18:15:30.634603 22482 tablet_bootstrap.cc:492] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed: No bootstrap required, opened a new log
I20260810 18:15:30.634809 22482 ts_tablet_manager.cc:1397] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:15:30.635462 22483 tablet_bootstrap.cc:492] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: No bootstrap required, opened a new log
I20260810 18:15:30.635666 22483 ts_tablet_manager.cc:1397] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:15:30.637017 22481 raft_consensus.cc:348] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } }
I20260810 18:15:30.637249 22481 raft_consensus.cc:374] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:30.637303 22481 raft_consensus.cc:729] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fe34f5401946dbbe5c2b5b86f73623, State: Initialized, Role: FOLLOWER
I20260810 18:15:30.637712 22481 consensus_queue.cc:260] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } }
I20260810 18:15:30.638381 22269 heartbeater.cc:503] Master 127.23.65.126:40797 was elected leader, sending a full tablet report...
I20260810 18:15:30.638736 22481 ts_tablet_manager.cc:1428] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:15:30.639605 22482 raft_consensus.cc:348] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } }
I20260810 18:15:30.639883 22482 raft_consensus.cc:374] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:30.639720 22483 raft_consensus.cc:348] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } }
I20260810 18:15:30.639945 22482 raft_consensus.cc:729] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177dc272268040a0ad9e31b89dd35bed, State: Initialized, Role: FOLLOWER
I20260810 18:15:30.639954 22483 raft_consensus.cc:374] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:30.639997 22483 raft_consensus.cc:729] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7658e306dd460395e7efe9e4d4a324, State: Initialized, Role: FOLLOWER
I20260810 18:15:30.640383 22482 consensus_queue.cc:260] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } }
I20260810 18:15:30.640383 22483 consensus_queue.cc:260] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } }
I20260810 18:15:30.641278 22045 heartbeater.cc:503] Master 127.23.65.126:40797 was elected leader, sending a full tablet report...
I20260810 18:15:30.641346 22482 ts_tablet_manager.cc:1428] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:15:30.641346 22483 ts_tablet_manager.cc:1428] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260810 18:15:30.641925 22270 tablet.cc:2369] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:15:30.642257 22457 heartbeater.cc:503] Master 127.23.65.126:40797 was elected leader, sending a full tablet report...
W20260810 18:15:30.681553 22047 tablet.cc:2369] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:15:30.827704 22458 tablet.cc:2369] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:15:30.921248 22489 raft_consensus.cc:482] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:30.921473 22489 raft_consensus.cc:504] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } }
I20260810 18:15:30.922792 22489 leader_election.cc:288] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 177dc272268040a0ad9e31b89dd35bed (127.23.65.65:37557), 07fe34f5401946dbbe5c2b5b86f73623 (127.23.65.66:41253)
I20260810 18:15:30.927212 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.923019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37557 (local address 127.23.65.67:34807)
0810 18:15:30.923617 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:30.923629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:30.923754 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:30.924005 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:30.924008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:30.924024 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:30.924135 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.924140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.924828 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.924831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.926165 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.926171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.926814 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.926818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.926911 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.926935 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:15:30.926992 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 18:15:30.927051 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":276,"thread_start_us":118,"threads_started":1}
I20260810 18:15:30.927243 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.923019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:41253 (local address 127.23.65.67:52879)
0810 18:15:30.923210 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:30.923227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:30.923333 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:30.923758 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:30.923761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:30.923780 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:30.923907 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.923913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.924976 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.924979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.926158 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:30.926170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.926881 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.926884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:30.926962 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.926981 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:15:30.927059 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:15:30.927127 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":31}
I20260810 18:15:30.927243 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.923278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:34807 (local address 127.23.65.65:37557)
0810 18:15:30.923409 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.923413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.923713 (+ 300us) server_negotiation.cc:407] Connection header received
0810 18:15:30.923860 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.923863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.923906 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.923967 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:30.924237 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.924715 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.926289 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.926690 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.926789 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.926845 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.927028 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.927113 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:15:30.927280 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.923214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:52879 (local address 127.23.65.66:41253)
0810 18:15:30.923506 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.923511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.923524 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:15:30.923599 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.923602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.923650 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.924008 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:30.924105 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.924875 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.926335 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.926729 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.926839 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.926888 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.927093 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.927166 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260810 18:15:30.927891 21980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813cd0760ce44ffac5752d088aca3ac" candidate_uuid: "ed7658e306dd460395e7efe9e4d4a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "177dc272268040a0ad9e31b89dd35bed" is_pre_election: true
I20260810 18:15:30.928122 22200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813cd0760ce44ffac5752d088aca3ac" candidate_uuid: "ed7658e306dd460395e7efe9e4d4a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fe34f5401946dbbe5c2b5b86f73623" is_pre_election: true
I20260810 18:15:30.928182 21980 raft_consensus.cc:2454] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed7658e306dd460395e7efe9e4d4a324 in term 0.
I20260810 18:15:30.928386 22200 raft_consensus.cc:2454] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed7658e306dd460395e7efe9e4d4a324 in term 0.
I20260810 18:15:30.928663 22294 leader_election.cc:302] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 177dc272268040a0ad9e31b89dd35bed, ed7658e306dd460395e7efe9e4d4a324; no voters:
I20260810 18:15:30.928933 22489 raft_consensus.cc:2781] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:15:30.929013 22489 raft_consensus.cc:482] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:15:30.929044 22489 raft_consensus.cc:3037] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:30.930068 22489 raft_consensus.cc:504] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } }
I20260810 18:15:30.930435 22489 leader_election.cc:288] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [CANDIDATE]: Term 1 election: Requested vote from peers 177dc272268040a0ad9e31b89dd35bed (127.23.65.65:37557), 07fe34f5401946dbbe5c2b5b86f73623 (127.23.65.66:41253)
I20260810 18:15:30.930766 21980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813cd0760ce44ffac5752d088aca3ac" candidate_uuid: "ed7658e306dd460395e7efe9e4d4a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "177dc272268040a0ad9e31b89dd35bed"
I20260810 18:15:30.930801 22200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2813cd0760ce44ffac5752d088aca3ac" candidate_uuid: "ed7658e306dd460395e7efe9e4d4a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fe34f5401946dbbe5c2b5b86f73623"
I20260810 18:15:30.930909 21980 raft_consensus.cc:3037] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:30.930932 22200 raft_consensus.cc:3037] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:30.932036 22200 raft_consensus.cc:2454] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed7658e306dd460395e7efe9e4d4a324 in term 1.
I20260810 18:15:30.932060 21980 raft_consensus.cc:2454] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed7658e306dd460395e7efe9e4d4a324 in term 1.
I20260810 18:15:30.932374 22294 leader_election.cc:302] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 177dc272268040a0ad9e31b89dd35bed, ed7658e306dd460395e7efe9e4d4a324; no voters:
I20260810 18:15:30.932552 22489 raft_consensus.cc:2781] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:30.932794 22489 raft_consensus.cc:686] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 1 LEADER]: Becoming Leader. State: Replica: ed7658e306dd460395e7efe9e4d4a324, State: Running, Role: LEADER
I20260810 18:15:30.933118 22489 consensus_queue.cc:237] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } }
I20260810 18:15:30.936244 21745 catalog_manager.cc:5495] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed7658e306dd460395e7efe9e4d4a324 (127.23.65.67). New cstate: current_term: 1 leader_uuid: "ed7658e306dd460395e7efe9e4d4a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed7658e306dd460395e7efe9e4d4a324" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 } health_report { overall_health: UNKNOWN } } }
I20260810 18:15:30.985128 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:30.981596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.23.65.67:46587)
0810 18:15:30.981721 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:30.981725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:30.981739 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:15:30.981868 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:30.981870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:30.981930 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:30.982022 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:15:30.982285 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.982778 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.983654 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:30.983876 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:30.983964 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:30.984272 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:15:30.984678 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:15:30.984822 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:30.984934 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 18:15:30.984969 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:15:30.990872 22200 raft_consensus.cc:1261] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [term 1 FOLLOWER]: Refusing update from remote peer ed7658e306dd460395e7efe9e4d4a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:30.990872 21980 raft_consensus.cc:1261] T 2813cd0760ce44ffac5752d088aca3ac P 177dc272268040a0ad9e31b89dd35bed [term 1 FOLLOWER]: Refusing update from remote peer ed7658e306dd460395e7efe9e4d4a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:30.991516 22497 consensus_queue.cc:1035] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "177dc272268040a0ad9e31b89dd35bed" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 37557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:30.991721 22489 consensus_queue.cc:1035] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07fe34f5401946dbbe5c2b5b86f73623" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:30.998382 22502 mvcc.cc:205] Tried to move back new op lower bound from 7317035954131079168 to 7317035953904394240. Current Snapshot: MvccSnapshot[applied={T|T < 7317035954131079168}]
I20260810 18:15:31.006953 21747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:31.007110 21747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:31.009495 21747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 236dbaef528b4d4eb607841103043c59: Sending DeleteTablet for 3 replicas of tablet 2813cd0760ce44ffac5752d088aca3ac
I20260810 18:15:31.010314 21942 tablet_service.cc:1514] Processing DeleteTablet for tablet 2813cd0760ce44ffac5752d088aca3ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:31 UTC) from {username='slave'} at 127.0.0.1:41508
I20260810 18:15:31.010365 22166 tablet_service.cc:1514] Processing DeleteTablet for tablet 2813cd0760ce44ffac5752d088aca3ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:31 UTC) from {username='slave'} at 127.0.0.1:39124
I20260810 18:15:31.011109 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 21842
I20260810 18:15:31.011094 22364 tablet_service.cc:1514] Processing DeleteTablet for tablet 2813cd0760ce44ffac5752d088aca3ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:31 UTC) from {username='slave'} at 127.0.0.1:58446
I20260810 18:15:31.018702 22520 tablet_replica.cc:330] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: stopping tablet replica
I20260810 18:15:31.019114 22520 raft_consensus.cc:2229] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:15:31.020094 22520 raft_consensus.cc:2258] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:31.020697 22522 tablet_replica.cc:330] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: stopping tablet replica
I20260810 18:15:31.020990 22522 raft_consensus.cc:2229] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:15:31.021341 22522 raft_consensus.cc:2258] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:31.021905 22520 ts_tablet_manager.cc:1905] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:31.022801 22522 ts_tablet_manager.cc:1905] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:31.024730 22520 ts_tablet_manager.cc:1918] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:31.024838 22520 log.cc:1196] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/2813cd0760ce44ffac5752d088aca3ac
I20260810 18:15:31.025166 22520 ts_tablet_manager.cc:1939] T 2813cd0760ce44ffac5752d088aca3ac P 07fe34f5401946dbbe5c2b5b86f73623: Deleting consensus metadata
I20260810 18:15:31.025453 22522 ts_tablet_manager.cc:1918] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:31.025540 22522 log.cc:1196] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/2813cd0760ce44ffac5752d088aca3ac
I20260810 18:15:31.025815 22522 ts_tablet_manager.cc:1939] T 2813cd0760ce44ffac5752d088aca3ac P ed7658e306dd460395e7efe9e4d4a324: Deleting consensus metadata
I20260810 18:15:31.026469 21729 catalog_manager.cc:4841] TS 07fe34f5401946dbbe5c2b5b86f73623 (127.23.65.66:41253): tablet 2813cd0760ce44ffac5752d088aca3ac (table test-table [id=2abb82d51bec4524b82c244210627220]) successfully deleted
I20260810 18:15:31.026893 21726 catalog_manager.cc:4841] TS ed7658e306dd460395e7efe9e4d4a324 (127.23.65.67:46587): tablet 2813cd0760ce44ffac5752d088aca3ac (table test-table [id=2abb82d51bec4524b82c244210627220]) successfully deleted
W20260810 18:15:31.027637 21726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:37557 (error 108)
I20260810 18:15:31.028465 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 22051
I20260810 18:15:31.028714 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:31.028401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:37557 (local address 127.0.0.1:36194)
0810 18:15:31.028558 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:31.028630 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 18:15:31.044121 21726 catalog_manager.cc:4584] TS 177dc272268040a0ad9e31b89dd35bed (127.23.65.65:37557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2813cd0760ce44ffac5752d088aca3ac: Network error: Client connection negotiation failed: client connection to 127.23.65.65:37557: connect: Connection refused (error 111)
I20260810 18:15:31.049515 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 22272
I20260810 18:15:31.057984 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 21699
2026-08-10T18:15:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1354 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:31 dist-test-slave-jpj1 krb5kdc[22527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:31 dist-test-slave-jpj1 krb5kdc[22527](info): set up 2 sockets
Aug 10 18:15:31 dist-test-slave-jpj1 krb5kdc[22527](info): commencing operation
krb5kdc: starting...
W20260810 18:15:32.214931 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:32 dist-test-slave-jpj1 krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:15:32.336035 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43157
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:32.454330 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:32.454617 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:32.454659 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:32.458664 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:32.458724 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:32.458743 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:32.458758 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:32.458774 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:32.464079 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:43157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:32.465142 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:32.466154 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:32.474359 22561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:32.474367 22555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:32.474375 22558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:32.474566 22544 server_base.cc:1034] running on GCE node
I20260810 18:15:32.475374 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:32.476504 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:32.477695 22544 hybrid_clock.cc:648] HybridClock initialized: now 1786385732477672 us; error 37 us; skew 500 ppm
Aug 10 18:15:32 dist-test-slave-jpj1 krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385732, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:32.480399 22544 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:15:32.481284 22544 webserver.cc:460] Webserver started at http://127.23.65.126:38275/ using document root <none> and password file <none>
I20260810 18:15:32.481841 22544 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:32.481913 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:32.482133 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:32.484302 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "593789ebfabd4070878cad0b95a40e39"
format_stamp: "Formatted at 2026-08-10 18:15:32 on dist-test-slave-jpj1"
server_key: "21b8864110201c6df933ee21d912bd58"
server_key_iv: "336097591de8e66b0f6591084abaed1e"
server_key_version: "encryptionkey@0"
I20260810 18:15:32.484844 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "593789ebfabd4070878cad0b95a40e39"
format_stamp: "Formatted at 2026-08-10 18:15:32 on dist-test-slave-jpj1"
server_key: "21b8864110201c6df933ee21d912bd58"
server_key_iv: "336097591de8e66b0f6591084abaed1e"
server_key_version: "encryptionkey@0"
I20260810 18:15:32.488456 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:32.490870 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:32.492084 22544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:32.492197 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "593789ebfabd4070878cad0b95a40e39"
format_stamp: "Formatted at 2026-08-10 18:15:32 on dist-test-slave-jpj1"
server_key: "21b8864110201c6df933ee21d912bd58"
server_key_iv: "336097591de8e66b0f6591084abaed1e"
server_key_version: "encryptionkey@0"
I20260810 18:15:32.492297 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:32.511291 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:32.514112 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:32.514289 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:32.521337 22544 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:43157
I20260810 18:15:32.521425 22672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:43157 every 8 connection(s)
I20260810 18:15:32.522351 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:32.522882 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 22544
I20260810 18:15:32.523036 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:32.523424 23813 external_mini_cluster.cc:1428] Setting key 0b92ac6b3a0a3647d319c40bf3389772
I20260810 18:15:32.526314 22674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 18:15:32 dist-test-slave-jpj1 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:32.532168 22674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39: Bootstrap starting.
I20260810 18:15:32.534451 22674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:32.535184 22674 log.cc:824] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:32.536873 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:32.524723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.23.65.126:43157)
0810 18:15:32.525130 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:32.525145 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:32.525183 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:15:32.525836 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:32.525853 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:32.526190 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:32.526513 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:32.527323 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:32.528390 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:32.529248 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:32.529750 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:32.529911 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:32.532527 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:32.532532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:32.532545 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:32.532573 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:32.534091 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:32.534520 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:32.534525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:32.534530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:32.534667 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:32.534938 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:32.534942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:32.534943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:32.535356 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:32.535565 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:32.535867 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 18:15:32.536154 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260810 18:15:32.537106 22674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39: No bootstrap required, opened a new log
I20260810 18:15:32.540689 22674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593789ebfabd4070878cad0b95a40e39" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43157 } }
I20260810 18:15:32.540974 22674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:32.541041 22674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593789ebfabd4070878cad0b95a40e39, State: Initialized, Role: FOLLOWER
I20260810 18:15:32.541515 22674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593789ebfabd4070878cad0b95a40e39" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43157 } }
I20260810 18:15:32.541683 22674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:32.541750 22674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:32.541850 22674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:32.542914 22674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593789ebfabd4070878cad0b95a40e39" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43157 } }
I20260810 18:15:32.543289 22674 leader_election.cc:302] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 593789ebfabd4070878cad0b95a40e39; no voters:
I20260810 18:15:32.543668 22674 leader_election.cc:288] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:32.543787 22680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:32.544032 22680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [term 1 LEADER]: Becoming Leader. State: Replica: 593789ebfabd4070878cad0b95a40e39, State: Running, Role: LEADER
I20260810 18:15:32.544332 22680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593789ebfabd4070878cad0b95a40e39" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43157 } }
I20260810 18:15:32.544889 22674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:32.545900 22687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 593789ebfabd4070878cad0b95a40e39. Latest consensus state: current_term: 1 leader_uuid: "593789ebfabd4070878cad0b95a40e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593789ebfabd4070878cad0b95a40e39" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43157 } } }
I20260810 18:15:32.546030 22687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:32.546312 22683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "593789ebfabd4070878cad0b95a40e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593789ebfabd4070878cad0b95a40e39" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43157 } } }
I20260810 18:15:32.546406 22683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:32.546509 22699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:32.549777 22699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:32.554988 22699 catalog_manager.cc:1347] Generated new cluster ID: c0d8654d26cc4548bd75d9fdfa379cac
I20260810 18:15:32.555099 22699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:32.569554 22699 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:32.570485 22699 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:32.576622 22699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39: Generated new TSK 0
I20260810 18:15:32.577258 22699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.65.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
Entry for principal kudu/127.23.65.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab.
I20260810 18:15:32.610241 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43157
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:32.722796 22717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:32.723057 22717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:32.723116 22717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:32.727093 22717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:32.727154 22717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:32.727237 22717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:15:32.732514 22717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.65.keytab
--principal=kudu/127.23.65.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:32.733557 22717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:32.734565 22717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:32.744071 22725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:32.744283 22717 server_base.cc:1034] running on GCE node
W20260810 18:15:32.744122 22729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:32.744086 22724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:32.744801 22717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:32.745448 22717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:32.746599 22717 hybrid_clock.cc:648] HybridClock initialized: now 1786385732746576 us; error 34 us; skew 500 ppm
Aug 10 18:15:32 dist-test-slave-jpj1 krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385732, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:32.749292 22717 init.cc:377] Logged in from keytab as kudu/127.23.65.65@KRBTEST.COM (short username kudu)
I20260810 18:15:32.750135 22717 webserver.cc:460] Webserver started at http://127.23.65.65:42619/ using document root <none> and password file <none>
I20260810 18:15:32.750679 22717 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:32.750742 22717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:32.750939 22717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:32.752962 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "815f99d296a74ab693ef5155bf18d4a5"
format_stamp: "Formatted at 2026-08-10 18:15:32 on dist-test-slave-jpj1"
server_key: "966642f88cacd2ea698bfe6c48b3c780"
server_key_iv: "ece38db220787fbfd074f766e669cfab"
server_key_version: "encryptionkey@0"
I20260810 18:15:32.753494 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "815f99d296a74ab693ef5155bf18d4a5"
format_stamp: "Formatted at 2026-08-10 18:15:32 on dist-test-slave-jpj1"
server_key: "966642f88cacd2ea698bfe6c48b3c780"
server_key_iv: "ece38db220787fbfd074f766e669cfab"
server_key_version: "encryptionkey@0"
I20260810 18:15:32.757058 22717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:15:32.759428 22738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:32.760633 22717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:32.760754 22717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "815f99d296a74ab693ef5155bf18d4a5"
format_stamp: "Formatted at 2026-08-10 18:15:32 on dist-test-slave-jpj1"
server_key: "966642f88cacd2ea698bfe6c48b3c780"
server_key_iv: "ece38db220787fbfd074f766e669cfab"
server_key_version: "encryptionkey@0"
I20260810 18:15:32.760872 22717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:32.795475 22717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:32.798753 22717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:32.798941 22717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:32.799553 22717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:32.800422 22717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:32.800470 22717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:32.800516 22717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:32.800545 22717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:32.808956 22717 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:43025
I20260810 18:15:32.809020 22953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:43025 every 8 connection(s)
I20260810 18:15:32.809875 22717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:15:32.817164 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 22717
I20260810 18:15:32.817301 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:15:32.817592 23813 external_mini_cluster.cc:1428] Setting key bc4c68d2a686f8c043a1d4466299edaa
Aug 10 18:15:32 dist-test-slave-jpj1 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385732, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.65@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:32.823249 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:32.811671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:46995 (local address 127.23.65.126:43157)
0810 18:15:32.811811 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:32.811815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:32.812622 (+ 807us) server_negotiation.cc:407] Connection header received
0810 18:15:32.813873 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:32.813877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:32.813933 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:32.814005 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:32.815169 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:32.815737 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:32.816854 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:32.817114 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:32.817202 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:32.819967 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:32.819970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:32.819972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:32.819995 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:32.821398 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:32.821858 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:32.821862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:32.821863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:32.821955 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:32.822234 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:32.822237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:32.822239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:32.822454 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:32.822548 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:32.822976 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 18:15:32.823096 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:15:32.823994 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:32.812036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43157 (local address 127.23.65.65:46995)
0810 18:15:32.812480 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:32.812522 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:32.813667 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:32.814184 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:32.814192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:32.814559 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:32.815021 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:32.815035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:32.815870 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:32.815873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:32.816735 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:32.816745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:32.817264 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:32.817269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:32.817451 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:32.817994 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:32.818013 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:32.819795 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:32.821515 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:32.821520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:32.821530 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:32.821771 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:32.822032 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:32.822034 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:32.822036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:32.822157 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:32.822539 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:32.822544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:32.822760 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 18:15:32.822904 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 18:15:32.823140 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260810 18:15:32.825208 22954 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43157
I20260810 18:15:32.825462 22954 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:32.826045 22954 heartbeater.cc:511] Master 127.23.65.126:43157 requested a full tablet report, sending...
I20260810 18:15:32.827564 22609 ts_manager.cc:194] Registered new tserver with Master: 815f99d296a74ab693ef5155bf18d4a5 (127.23.65.65:43025)
WARNING: no policy specified for kudu/127.23.65.66@KRBTEST.COM; defaulting to no policy
I20260810 18:15:32.828697 22609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.65@KRBTEST.COM'} at 127.23.65.65:46995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
Entry for principal kudu/127.23.65.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab.
I20260810 18:15:32.844224 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43157
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:32.957502 22966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:32.957773 22966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:32.957815 22966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:32.961804 22966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:32.961876 22966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:32.961939 22966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:15:32.967252 22966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.66.keytab
--principal=kudu/127.23.65.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:32.968386 22966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:32.969380 22966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:32.978732 22977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:32.978720 22972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:32.978811 22966 server_base.cc:1034] running on GCE node
W20260810 18:15:32.978698 22973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:32.979301 22966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:32.979954 22966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:32.981127 22966 hybrid_clock.cc:648] HybridClock initialized: now 1786385732981110 us; error 27 us; skew 500 ppm
Aug 10 18:15:32 dist-test-slave-jpj1 krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385732, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:32.983839 22966 init.cc:377] Logged in from keytab as kudu/127.23.65.66@KRBTEST.COM (short username kudu)
I20260810 18:15:32.984643 22966 webserver.cc:460] Webserver started at http://127.23.65.66:36623/ using document root <none> and password file <none>
I20260810 18:15:32.985244 22966 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:32.985309 22966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:32.985481 22966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:32.987495 22966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "f0daddcfab5f47108811f0ed5b17bf34"
format_stamp: "Formatted at 2026-08-10 18:15:32 on dist-test-slave-jpj1"
server_key: "f4d7288d753351d6e321dcebb5b21a8d"
server_key_iv: "c79a6d69319fe8999cf9385b323d26d9"
server_key_version: "encryptionkey@0"
I20260810 18:15:32.988063 22966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "f0daddcfab5f47108811f0ed5b17bf34"
format_stamp: "Formatted at 2026-08-10 18:15:32 on dist-test-slave-jpj1"
server_key: "f4d7288d753351d6e321dcebb5b21a8d"
server_key_iv: "c79a6d69319fe8999cf9385b323d26d9"
server_key_version: "encryptionkey@0"
I20260810 18:15:32.991765 22966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:15:32.994065 22986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:32.995224 22966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:32.995353 22966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "f0daddcfab5f47108811f0ed5b17bf34"
format_stamp: "Formatted at 2026-08-10 18:15:32 on dist-test-slave-jpj1"
server_key: "f4d7288d753351d6e321dcebb5b21a8d"
server_key_iv: "c79a6d69319fe8999cf9385b323d26d9"
server_key_version: "encryptionkey@0"
I20260810 18:15:32.995448 22966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:33.016644 22966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:33.019621 22966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:33.019812 22966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:33.020440 22966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:33.021313 22966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:33.021365 22966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:33.021421 22966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:33.021452 22966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:33.030174 22966 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:45893
I20260810 18:15:33.030205 23164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:45893 every 8 connection(s)
I20260810 18:15:33.031067 22966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:15:33.040186 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 22966
I20260810 18:15:33.040324 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
Aug 10 18:15:33 dist-test-slave-jpj1 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385732, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.66@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:33.040650 23813 external_mini_cluster.cc:1428] Setting key defd02a75f197bfcc90bf6c19f9830a7
I20260810 18:15:33.044983 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.032817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:43429 (local address 127.23.65.126:43157)
0810 18:15:33.032937 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.032941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.033755 (+ 814us) server_negotiation.cc:407] Connection header received
0810 18:15:33.035041 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.035044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.035106 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.035181 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:33.036559 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.037090 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.038223 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.038446 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.038538 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.041272 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:33.041276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:33.041279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:33.041309 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:33.042842 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:33.043335 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:33.043338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:33.043340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:33.043433 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:33.043773 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:33.043776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:33.043778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:33.043992 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:33.044099 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.044652 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.044811 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:15:33.045778 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.033148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43157 (local address 127.23.65.66:43429)
0810 18:15:33.033601 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:33.033646 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:33.034813 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:33.035378 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:33.035386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:33.035794 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:33.036404 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.036425 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.037214 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.037217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.038100 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.038110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.038594 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.038598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.038760 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.039283 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:33.039300 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:33.041040 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:33.042968 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:33.042974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:33.042986 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:33.043242 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:33.043509 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:33.043511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:33.043512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:33.043689 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:33.044079 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:33.044085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:33.044336 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 18:15:33.044537 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 18:15:33.044865 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260810 18:15:33.046970 23165 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43157
I20260810 18:15:33.047201 23165 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:33.047715 23165 heartbeater.cc:511] Master 127.23.65.126:43157 requested a full tablet report, sending...
I20260810 18:15:33.048731 22609 ts_manager.cc:194] Registered new tserver with Master: f0daddcfab5f47108811f0ed5b17bf34 (127.23.65.66:45893)
I20260810 18:15:33.049288 22609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.66@KRBTEST.COM'} at 127.23.65.66:43429
WARNING: no policy specified for kudu/127.23.65.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
Entry for principal kudu/127.23.65.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab.
I20260810 18:15:33.067871 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43157
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:33.181857 23170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:33.182133 23170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:33.182176 23170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:33.186381 23170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:33.186462 23170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:33.186548 23170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:15:33.192092 23170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.67.keytab
--principal=kudu/127.23.65.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:33.193182 23170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:33.194231 23170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:33.203892 23179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:33.203953 23170 server_base.cc:1034] running on GCE node
W20260810 18:15:33.203884 23176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:33.203881 23177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:33.204582 23170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:33.205209 23170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:33.206370 23170 hybrid_clock.cc:648] HybridClock initialized: now 1786385733206343 us; error 43 us; skew 500 ppm
Aug 10 18:15:33 dist-test-slave-jpj1 krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385733, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:33.209069 23170 init.cc:377] Logged in from keytab as kudu/127.23.65.67@KRBTEST.COM (short username kudu)
I20260810 18:15:33.210012 23170 webserver.cc:460] Webserver started at http://127.23.65.67:44483/ using document root <none> and password file <none>
I20260810 18:15:33.210575 23170 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:33.210644 23170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:33.210852 23170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:33.212975 23170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "3745d3eb191c4417887fdf086dd07bbc"
format_stamp: "Formatted at 2026-08-10 18:15:33 on dist-test-slave-jpj1"
server_key: "9a2dce21832a5915df419e2ee64f1e2c"
server_key_iv: "19f04d7bef4797bb20fd97baafd22b48"
server_key_version: "encryptionkey@0"
I20260810 18:15:33.213526 23170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "3745d3eb191c4417887fdf086dd07bbc"
format_stamp: "Formatted at 2026-08-10 18:15:33 on dist-test-slave-jpj1"
server_key: "9a2dce21832a5915df419e2ee64f1e2c"
server_key_iv: "19f04d7bef4797bb20fd97baafd22b48"
server_key_version: "encryptionkey@0"
I20260810 18:15:33.217306 23170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 18:15:33.219676 23203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:33.220847 23170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:15:33.220968 23170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "3745d3eb191c4417887fdf086dd07bbc"
format_stamp: "Formatted at 2026-08-10 18:15:33 on dist-test-slave-jpj1"
server_key: "9a2dce21832a5915df419e2ee64f1e2c"
server_key_iv: "19f04d7bef4797bb20fd97baafd22b48"
server_key_version: "encryptionkey@0"
I20260810 18:15:33.221060 23170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:33.238481 23170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:33.241308 23170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:33.241492 23170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:33.242089 23170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:33.242924 23170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:33.242974 23170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:33.243028 23170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:33.243055 23170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:33.252086 23170 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:45979
I20260810 18:15:33.252111 23412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:45979 every 8 connection(s)
I20260810 18:15:33.253000 23170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:15:33.254411 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 23170
I20260810 18:15:33.254526 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:15:33.254822 23813 external_mini_cluster.cc:1428] Setting key b007e40ba900733ff56bb404cc653406
Aug 10 18:15:33 dist-test-slave-jpj1 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385733, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.67@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:33.266304 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.254722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:38453 (local address 127.23.65.126:43157)
0810 18:15:33.254876 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.254881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.255976 (+ 1095us) server_negotiation.cc:407] Connection header received
0810 18:15:33.257399 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.257402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.257456 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.257526 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:33.258655 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.259286 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.260430 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.260651 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.260738 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.263097 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:33.263101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:33.263103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:33.263128 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:33.264394 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:33.264857 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:33.264861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:33.264862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:33.264941 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:33.265262 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:33.265265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:33.265267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:33.265460 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:33.265549 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.266020 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.266141 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:15:33.267220 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.255189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43157 (local address 127.23.65.67:38453)
0810 18:15:33.255833 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:33.255889 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:33.257205 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:33.257693 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:33.257699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:33.258047 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:33.258521 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.258536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.259404 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.259408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.260317 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.260328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.260786 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.260790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.260957 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.261482 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:15:33.261498 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:33.262931 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:33.264493 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:33.264499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:33.264514 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:33.264763 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:33.265022 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:15:33.265025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:15:33.265027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:15:33.265162 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:15:33.265550 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:33.265557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:33.265797 (+ 240us) client_negotiation.cc:769] Sending connection context
0810 18:15:33.265961 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 18:15:33.266282 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":135,"threads_started":1}
I20260810 18:15:33.268241 23414 heartbeater.cc:347] Connected to a master server at 127.23.65.126:43157
I20260810 18:15:33.268493 23414 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:33.269011 23414 heartbeater.cc:511] Master 127.23.65.126:43157 requested a full tablet report, sending...
I20260810 18:15:33.269966 22609 ts_manager.cc:194] Registered new tserver with Master: 3745d3eb191c4417887fdf086dd07bbc (127.23.65.67:45979)
I20260810 18:15:33.270479 22609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.65.67@KRBTEST.COM'} at 127.23.65.67:38453
I20260810 18:15:33.280278 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:15:33.289065 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.282257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.23.65.126:43157)
0810 18:15:33.282370 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.282374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.282569 (+ 195us) server_negotiation.cc:407] Connection header received
0810 18:15:33.282686 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.282689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.282730 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.282791 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:33.283497 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.284047 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.284867 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.285083 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.285171 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.286429 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:33.286433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:33.286436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:33.286460 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:33.287492 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:33.287899 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:33.287903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:33.287904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:33.287982 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:33.288227 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:33.288231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:33.288232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:33.288449 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:33.288569 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.288793 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.288910 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 18:15:33.291473 22609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:15:33.294528 22609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:15:33.308564 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.302761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:43025 (local address 127.0.0.1:33338)
0810 18:15:33.303098 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:33.303142 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:33.303273 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:33.303867 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:33.303873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:33.303896 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:33.304048 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.304057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.305396 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.305399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.306701 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.306711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.308021 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.308025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.308137 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.308176 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 18:15:33.308287 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 18:15:33.308409 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260810 18:15:33.308740 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.303882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:45893 (local address 127.0.0.1:59458)
0810 18:15:33.304095 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:33.304109 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:33.304189 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:33.305137 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:33.305140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:33.305157 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:33.305241 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.305258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.306340 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.306343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.307552 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.307563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.308401 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.308405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.308486 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.308511 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:15:33.308556 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 18:15:33.308623 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":12,"thread_start_us":96,"threads_started":1}
I20260810 18:15:33.308620 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.302966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.23.65.65:43025)
0810 18:15:33.303332 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.303350 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.303374 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:15:33.303450 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.303456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.303709 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.303866 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:33.304185 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.305272 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.306822 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.307685 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.307784 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.307950 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.308345 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.308451 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":147,"threads_started":1}
I20260810 18:15:33.309177 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.303518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:45979 (local address 127.0.0.1:44354)
0810 18:15:33.303773 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:33.303786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:33.303888 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:33.305025 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:33.305028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:33.305044 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:33.305144 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.305151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.306505 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.306508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.307674 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.307682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.308818 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.308822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.308917 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.308941 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:15:33.308990 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 18:15:33.309056 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":40,"thread_start_us":99,"threads_started":1}
I20260810 18:15:33.309273 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.304275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.23.65.66:45893)
0810 18:15:33.304710 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.304720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.304742 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:15:33.304816 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.304822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.304993 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.305398 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:33.305534 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.306202 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.307693 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.308262 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.308909 (+ 647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.309047 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.309089 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.309158 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":129,"threads_started":1}
I20260810 18:15:33.310014 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.304212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.23.65.67:45979)
0810 18:15:33.304554 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.304566 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.304589 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:15:33.304662 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.304667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.304883 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.305044 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:33.305237 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.306374 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.307771 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.308593 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.309571 (+ 978us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.309736 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.309789 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.309878 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":143,"threads_started":1}
I20260810 18:15:33.311939 22827 tablet_service.cc:1467] Processing CreateTablet for tablet ccdd764bef774543b26115e7ee60e673 (DEFAULT_TABLE table=test-table [id=2de7caef46fd42439f00721a71835559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:33.311944 23060 tablet_service.cc:1467] Processing CreateTablet for tablet ccdd764bef774543b26115e7ee60e673 (DEFAULT_TABLE table=test-table [id=2de7caef46fd42439f00721a71835559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:33.312889 22827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdd764bef774543b26115e7ee60e673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:33.312513 23296 tablet_service.cc:1467] Processing CreateTablet for tablet ccdd764bef774543b26115e7ee60e673 (DEFAULT_TABLE table=test-table [id=2de7caef46fd42439f00721a71835559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:33.313004 23060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdd764bef774543b26115e7ee60e673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:33.313411 23296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdd764bef774543b26115e7ee60e673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:33.318881 23438 tablet_bootstrap.cc:492] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5: Bootstrap starting.
I20260810 18:15:33.320433 23441 tablet_bootstrap.cc:492] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: Bootstrap starting.
I20260810 18:15:33.320788 23439 tablet_bootstrap.cc:492] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: Bootstrap starting.
I20260810 18:15:33.321662 23438 tablet_bootstrap.cc:654] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:33.322403 23439 tablet_bootstrap.cc:654] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:33.322691 23438 log.cc:824] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:33.323459 23439 log.cc:824] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:33.324911 23438 tablet_bootstrap.cc:492] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5: No bootstrap required, opened a new log
I20260810 18:15:33.324960 23441 tablet_bootstrap.cc:654] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:33.325295 23438 ts_tablet_manager.cc:1397] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260810 18:15:33.325387 23439 tablet_bootstrap.cc:492] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: No bootstrap required, opened a new log
I20260810 18:15:33.325559 23439 ts_tablet_manager.cc:1397] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:15:33.326093 23441 log.cc:824] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:33.328034 23441 tablet_bootstrap.cc:492] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: No bootstrap required, opened a new log
I20260810 18:15:33.328214 23441 ts_tablet_manager.cc:1397] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 18:15:33.328608 23439 raft_consensus.cc:348] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } }
I20260810 18:15:33.328837 23439 raft_consensus.cc:374] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:33.328891 23439 raft_consensus.cc:729] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0daddcfab5f47108811f0ed5b17bf34, State: Initialized, Role: FOLLOWER
I20260810 18:15:33.329094 23438 raft_consensus.cc:348] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } }
I20260810 18:15:33.329276 23439 consensus_queue.cc:260] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } }
I20260810 18:15:33.329345 23438 raft_consensus.cc:374] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:33.329406 23438 raft_consensus.cc:729] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815f99d296a74ab693ef5155bf18d4a5, State: Initialized, Role: FOLLOWER
I20260810 18:15:33.329895 23165 heartbeater.cc:503] Master 127.23.65.126:43157 was elected leader, sending a full tablet report...
I20260810 18:15:33.329897 23438 consensus_queue.cc:260] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } }
I20260810 18:15:33.330505 22954 heartbeater.cc:503] Master 127.23.65.126:43157 was elected leader, sending a full tablet report...
I20260810 18:15:33.330888 23438 ts_tablet_manager.cc:1428] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260810 18:15:33.331285 23439 ts_tablet_manager.cc:1428] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:15:33.332778 23441 raft_consensus.cc:348] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } }
I20260810 18:15:33.333081 23441 raft_consensus.cc:374] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:33.333151 23441 raft_consensus.cc:729] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3745d3eb191c4417887fdf086dd07bbc, State: Initialized, Role: FOLLOWER
I20260810 18:15:33.333545 23441 consensus_queue.cc:260] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } }
I20260810 18:15:33.334149 23414 heartbeater.cc:503] Master 127.23.65.126:43157 was elected leader, sending a full tablet report...
I20260810 18:15:33.334471 23441 ts_tablet_manager.cc:1428] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:15:33.481853 23445 raft_consensus.cc:482] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:33.482112 23445 raft_consensus.cc:504] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } }
I20260810 18:15:33.483306 23445 leader_election.cc:288] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 815f99d296a74ab693ef5155bf18d4a5 (127.23.65.65:43025), 3745d3eb191c4417887fdf086dd07bbc (127.23.65.67:45979)
I20260810 18:15:33.488247 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.484045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:44717 (local address 127.23.65.67:45979)
0810 18:15:33.484184 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.484188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.484345 (+ 157us) server_negotiation.cc:407] Connection header received
0810 18:15:33.484455 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.484457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.484495 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.484549 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:33.484800 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.485493 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.487006 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.487638 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.487783 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.487838 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.488054 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.488132 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:15:33.488263 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.483876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:45979 (local address 127.23.65.66:44717)
0810 18:15:33.484251 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:33.484264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:33.484363 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:33.484577 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:33.484579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:33.484595 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:33.484690 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.484695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.485578 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.485581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.486886 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.486897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.487753 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.487756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.487866 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.487892 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:15:33.488005 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 18:15:33.488109 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":83,"thread_start_us":156,"threads_started":1}
I20260810 18:15:33.488507 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.483693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:57259 (local address 127.23.65.65:43025)
0810 18:15:33.483841 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.483845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.483863 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:15:33.484171 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.484174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.484228 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.484303 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:33.484607 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.485402 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.487008 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.487648 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.487787 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.487838 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.488319 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.488395 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:15:33.488763 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.483599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:43025 (local address 127.23.65.66:57259)
0810 18:15:33.483736 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:33.483753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:33.483870 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:33.484351 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:33.484354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:33.484368 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:33.484501 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.484506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.485507 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.485510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.486885 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:33.486897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.487895 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.487898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:33.488013 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.488060 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 18:15:33.488125 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:15:33.488199 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":14}
I20260810 18:15:33.489209 23330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdd764bef774543b26115e7ee60e673" candidate_uuid: "f0daddcfab5f47108811f0ed5b17bf34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3745d3eb191c4417887fdf086dd07bbc" is_pre_election: true
I20260810 18:15:33.489352 22863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdd764bef774543b26115e7ee60e673" candidate_uuid: "f0daddcfab5f47108811f0ed5b17bf34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "815f99d296a74ab693ef5155bf18d4a5" is_pre_election: true
I20260810 18:15:33.489553 23330 raft_consensus.cc:2454] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0daddcfab5f47108811f0ed5b17bf34 in term 0.
I20260810 18:15:33.489599 22863 raft_consensus.cc:2454] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0daddcfab5f47108811f0ed5b17bf34 in term 0.
I20260810 18:15:33.490046 22987 leader_election.cc:302] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3745d3eb191c4417887fdf086dd07bbc, f0daddcfab5f47108811f0ed5b17bf34; no voters:
I20260810 18:15:33.490310 23445 raft_consensus.cc:2781] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:15:33.490401 23445 raft_consensus.cc:482] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:15:33.490451 23445 raft_consensus.cc:3037] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:33.491506 23445 raft_consensus.cc:504] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } }
I20260810 18:15:33.491851 23445 leader_election.cc:288] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [CANDIDATE]: Term 1 election: Requested vote from peers 815f99d296a74ab693ef5155bf18d4a5 (127.23.65.65:43025), 3745d3eb191c4417887fdf086dd07bbc (127.23.65.67:45979)
I20260810 18:15:33.492165 22863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdd764bef774543b26115e7ee60e673" candidate_uuid: "f0daddcfab5f47108811f0ed5b17bf34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "815f99d296a74ab693ef5155bf18d4a5"
I20260810 18:15:33.492235 23330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdd764bef774543b26115e7ee60e673" candidate_uuid: "f0daddcfab5f47108811f0ed5b17bf34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3745d3eb191c4417887fdf086dd07bbc"
I20260810 18:15:33.492286 22863 raft_consensus.cc:3037] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:33.492362 23330 raft_consensus.cc:3037] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:33.493566 23330 raft_consensus.cc:2454] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0daddcfab5f47108811f0ed5b17bf34 in term 1.
I20260810 18:15:33.493584 22863 raft_consensus.cc:2454] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0daddcfab5f47108811f0ed5b17bf34 in term 1.
I20260810 18:15:33.493841 22987 leader_election.cc:302] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 815f99d296a74ab693ef5155bf18d4a5, f0daddcfab5f47108811f0ed5b17bf34; no voters:
I20260810 18:15:33.493986 23445 raft_consensus.cc:2781] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:33.494179 23445 raft_consensus.cc:686] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 1 LEADER]: Becoming Leader. State: Replica: f0daddcfab5f47108811f0ed5b17bf34, State: Running, Role: LEADER
I20260810 18:15:33.494478 23445 consensus_queue.cc:237] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } }
I20260810 18:15:33.497645 22607 catalog_manager.cc:5495] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0daddcfab5f47108811f0ed5b17bf34 (127.23.65.66). New cstate: current_term: 1 leader_uuid: "f0daddcfab5f47108811f0ed5b17bf34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0daddcfab5f47108811f0ed5b17bf34" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 45893 } health_report { overall_health: HEALTHY } } }
W20260810 18:15:33.503911 23415 tablet.cc:2369] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:15:33.532249 23166 tablet.cc:2369] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:15:33.541775 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.538629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.23.65.66:45893)
0810 18:15:33.538770 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.538773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.538786 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:15:33.538877 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.538879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.538935 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.539021 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:15:33.539199 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.539725 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.540537 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.540737 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.540814 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.541036 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:15:33.541469 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:15:33.541576 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.541603 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.541653 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:15:33.548194 22863 raft_consensus.cc:1261] T ccdd764bef774543b26115e7ee60e673 P 815f99d296a74ab693ef5155bf18d4a5 [term 1 FOLLOWER]: Refusing update from remote peer f0daddcfab5f47108811f0ed5b17bf34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:33.548194 23330 raft_consensus.cc:1261] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [term 1 FOLLOWER]: Refusing update from remote peer f0daddcfab5f47108811f0ed5b17bf34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:33.548902 23445 consensus_queue.cc:1035] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "815f99d296a74ab693ef5155bf18d4a5" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:33.549163 23453 consensus_queue.cc:1035] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3745d3eb191c4417887fdf086dd07bbc" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:33.557277 23464 mvcc.cc:205] Tried to move back new op lower bound from 7317035964605480960 to 7317035964395958272. Current Snapshot: MvccSnapshot[applied={T|T < 7317035964605480960}]
I20260810 18:15:33.565447 22609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:33.565601 22609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:33.567844 22609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 593789ebfabd4070878cad0b95a40e39: Sending DeleteTablet for 3 replicas of tablet ccdd764bef774543b26115e7ee60e673
I20260810 18:15:33.568678 23060 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdd764bef774543b26115e7ee60e673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:33 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:59458
I20260810 18:15:33.568828 22827 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdd764bef774543b26115e7ee60e673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:33 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:33338
I20260810 18:15:33.569211 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 22717
I20260810 18:15:33.569170 23296 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdd764bef774543b26115e7ee60e673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:33 UTC) from {username='kudu', principal='kudu/127.23.65.126@KRBTEST.COM'} at 127.0.0.1:44354
I20260810 18:15:33.569841 23474 tablet_replica.cc:330] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: stopping tablet replica
I20260810 18:15:33.573580 23474 raft_consensus.cc:2229] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:15:33.573946 23474 raft_consensus.cc:2258] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:33.581426 23478 tablet_replica.cc:330] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: stopping tablet replica
I20260810 18:15:33.581841 23478 raft_consensus.cc:2229] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:15:33.582252 23478 raft_consensus.cc:2258] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:33.583236 23474 ts_tablet_manager.cc:1905] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:33.583992 23478 ts_tablet_manager.cc:1905] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:33.586094 23474 ts_tablet_manager.cc:1918] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:33.586197 23474 log.cc:1196] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/ccdd764bef774543b26115e7ee60e673
I20260810 18:15:33.586493 23474 ts_tablet_manager.cc:1939] T ccdd764bef774543b26115e7ee60e673 P f0daddcfab5f47108811f0ed5b17bf34: Deleting consensus metadata
I20260810 18:15:33.587116 23478 ts_tablet_manager.cc:1918] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:33.587273 23478 log.cc:1196] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/ccdd764bef774543b26115e7ee60e673
I20260810 18:15:33.587657 23478 ts_tablet_manager.cc:1939] T ccdd764bef774543b26115e7ee60e673 P 3745d3eb191c4417887fdf086dd07bbc: Deleting consensus metadata
I20260810 18:15:33.587749 22581 catalog_manager.cc:4841] TS f0daddcfab5f47108811f0ed5b17bf34 (127.23.65.66:45893): tablet ccdd764bef774543b26115e7ee60e673 (table test-table [id=2de7caef46fd42439f00721a71835559]) successfully deleted
I20260810 18:15:33.588932 22581 catalog_manager.cc:4841] TS 3745d3eb191c4417887fdf086dd07bbc (127.23.65.67:45979): tablet ccdd764bef774543b26115e7ee60e673 (table test-table [id=2de7caef46fd42439f00721a71835559]) successfully deleted
W20260810 18:15:33.589923 22581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:43025 (error 108)
I20260810 18:15:33.589982 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 22966
I20260810 18:15:33.593020 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.592699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:43025 (local address 127.0.0.1:33340)
0810 18:15:33.592845 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:33.592927 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:15:33.593205 22581 catalog_manager.cc:4584] TS 815f99d296a74ab693ef5155bf18d4a5 (127.23.65.65:43025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccdd764bef774543b26115e7ee60e673: Network error: Client connection negotiation failed: client connection to 127.23.65.65:43025: connect: Connection refused (error 111)
I20260810 18:15:33.600189 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 23170
I20260810 18:15:33.613150 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 22544
2026-08-10T18:15:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2555 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T18:15:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:33Z Disabled control of system clock
I20260810 18:15:33.650954 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40155
--webserver_interface=127.23.65.126
--webserver_port=0
--builtin_ntp_servers=127.23.65.84:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:15:33.760821 23485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:33.761067 23485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:33.761101 23485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:33.764976 23485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:33.765028 23485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:33.765043 23485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:33.765056 23485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:33.765069 23485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:33.769718 23485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:40155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:33.770656 23485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:33.771987 23485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:33.779843 23496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:33.779861 23491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:33.780056 23485 server_base.cc:1034] running on GCE node
W20260810 18:15:33.780156 23493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:33.780663 23485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:33.781677 23485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:33.782830 23485 hybrid_clock.cc:648] HybridClock initialized: now 1786385733782808 us; error 38 us; skew 500 ppm
I20260810 18:15:33.784622 23485 webserver.cc:460] Webserver started at http://127.23.65.126:43917/ using document root <none> and password file <none>
I20260810 18:15:33.785149 23485 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:33.785216 23485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:33.785403 23485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:33.787460 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "77e803b2400149029a73e3eca1ae938f"
format_stamp: "Formatted at 2026-08-10 18:15:33 on dist-test-slave-jpj1"
server_key: "85fffef396fdacb609a0f3a057e925f0"
server_key_iv: "b80be3a8715e7e6021ec13f4318100fc"
server_key_version: "encryptionkey@0"
I20260810 18:15:33.788024 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "77e803b2400149029a73e3eca1ae938f"
format_stamp: "Formatted at 2026-08-10 18:15:33 on dist-test-slave-jpj1"
server_key: "85fffef396fdacb609a0f3a057e925f0"
server_key_iv: "b80be3a8715e7e6021ec13f4318100fc"
server_key_version: "encryptionkey@0"
I20260810 18:15:33.791411 23485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:15:33.793648 23506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:33.794801 23485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:33.794924 23485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "77e803b2400149029a73e3eca1ae938f"
format_stamp: "Formatted at 2026-08-10 18:15:33 on dist-test-slave-jpj1"
server_key: "85fffef396fdacb609a0f3a057e925f0"
server_key_iv: "b80be3a8715e7e6021ec13f4318100fc"
server_key_version: "encryptionkey@0"
I20260810 18:15:33.795013 23485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:33.811318 23485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:33.811964 23485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:33.812131 23485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:33.818958 23485 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:40155
I20260810 18:15:33.818974 23591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:40155 every 8 connection(s)
I20260810 18:15:33.819921 23485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:33.822923 23592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:33.826469 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 23485
I20260810 18:15:33.826591 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:33.826853 23813 external_mini_cluster.cc:1428] Setting key afd5d4d9bcd7869c238ad98a7dc30fda
I20260810 18:15:33.828698 23592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f: Bootstrap starting.
I20260810 18:15:33.831014 23592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:33.831799 23592 log.cc:824] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:33.833808 23592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f: No bootstrap required, opened a new log
I20260810 18:15:33.835640 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:33.828180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.23.65.126:40155)
0810 18:15:33.828806 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:33.828825 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:33.828871 (+ 46us) server_negotiation.cc:407] Connection header received
0810 18:15:33.829506 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:33.829523 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:33.829931 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:33.830194 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:33.830531 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.831519 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.832590 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:33.833051 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:33.833164 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:33.833980 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:33.833985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:33.833996 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:33.834365 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:33.834420 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:33.834440 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:33.834513 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:33.834667 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 18:15:33.834922 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260810 18:15:33.837452 23592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e803b2400149029a73e3eca1ae938f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40155 } }
I20260810 18:15:33.837666 23592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:33.837731 23592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e803b2400149029a73e3eca1ae938f, State: Initialized, Role: FOLLOWER
I20260810 18:15:33.838112 23592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e803b2400149029a73e3eca1ae938f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40155 } }
I20260810 18:15:33.838238 23592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:33.838300 23592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:33.838407 23592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:33.839226 23592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e803b2400149029a73e3eca1ae938f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40155 } }
I20260810 18:15:33.839566 23592 leader_election.cc:302] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77e803b2400149029a73e3eca1ae938f; no voters:
I20260810 18:15:33.839857 23592 leader_election.cc:288] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:33.839962 23601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:33.840183 23601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [term 1 LEADER]: Becoming Leader. State: Replica: 77e803b2400149029a73e3eca1ae938f, State: Running, Role: LEADER
I20260810 18:15:33.840487 23601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e803b2400149029a73e3eca1ae938f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40155 } }
I20260810 18:15:33.841037 23592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:33.841527 23603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77e803b2400149029a73e3eca1ae938f. Latest consensus state: current_term: 1 leader_uuid: "77e803b2400149029a73e3eca1ae938f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e803b2400149029a73e3eca1ae938f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40155 } } }
I20260810 18:15:33.841655 23603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:33.842033 23602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77e803b2400149029a73e3eca1ae938f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e803b2400149029a73e3eca1ae938f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 40155 } } }
I20260810 18:15:33.842131 23602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:33.842222 23607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:33.845229 23607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:33.850956 23607 catalog_manager.cc:1347] Generated new cluster ID: 818cf0f161dd42e2abc049e39bd48e50
I20260810 18:15:33.851008 23607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:33.879585 23607 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:33.880736 23607 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:33.887179 23607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f: Generated new TSK 0
I20260810 18:15:33.888067 23607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:33.894783 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.65:0
--local_ip_for_outbound_sockets=127.23.65.65
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40155
--builtin_ntp_servers=127.23.65.84:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:15:34.006392 23628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:34.006675 23628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:34.006716 23628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:34.010679 23628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:34.010735 23628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:34.010793 23628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.65
I20260810 18:15:34.015460 23628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.65.65
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.65
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:34.016558 23628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:34.017908 23628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:34.027554 23645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:34.027577 23637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:34.027604 23628 server_base.cc:1034] running on GCE node
W20260810 18:15:34.027627 23641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:34.028160 23628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:34.028853 23628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:34.030017 23628 hybrid_clock.cc:648] HybridClock initialized: now 1786385734029993 us; error 38 us; skew 500 ppm
I20260810 18:15:34.031993 23628 webserver.cc:460] Webserver started at http://127.23.65.65:37769/ using document root <none> and password file <none>
I20260810 18:15:34.032567 23628 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:34.032632 23628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:34.032857 23628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:34.034996 23628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/data/instance:
uuid: "ad70b297567c4357a45aa39cda7d7c3f"
format_stamp: "Formatted at 2026-08-10 18:15:34 on dist-test-slave-jpj1"
server_key: "ba321eb312c0508afc5304992af0c6d4"
server_key_iv: "f7316ca9aaa7b1f010a5ac7f5f928f6b"
server_key_version: "encryptionkey@0"
I20260810 18:15:34.035601 23628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance:
uuid: "ad70b297567c4357a45aa39cda7d7c3f"
format_stamp: "Formatted at 2026-08-10 18:15:34 on dist-test-slave-jpj1"
server_key: "ba321eb312c0508afc5304992af0c6d4"
server_key_iv: "f7316ca9aaa7b1f010a5ac7f5f928f6b"
server_key_version: "encryptionkey@0"
I20260810 18:15:34.039454 23628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:15:34.041983 23655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:34.043192 23628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:15:34.043323 23628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/wal
uuid: "ad70b297567c4357a45aa39cda7d7c3f"
format_stamp: "Formatted at 2026-08-10 18:15:34 on dist-test-slave-jpj1"
server_key: "ba321eb312c0508afc5304992af0c6d4"
server_key_iv: "f7316ca9aaa7b1f010a5ac7f5f928f6b"
server_key_version: "encryptionkey@0"
I20260810 18:15:34.043435 23628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:34.061946 23628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:34.062594 23628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:34.062762 23628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:34.063333 23628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:34.064447 23628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:34.064498 23628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:34.064536 23628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:34.064556 23628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:34.073411 23628 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.65:38891
I20260810 18:15:34.073432 23854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.65:38891 every 8 connection(s)
I20260810 18:15:34.074324 23628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/data/info.pb
I20260810 18:15:34.082187 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 23628
I20260810 18:15:34.082329 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-0/wal/instance
I20260810 18:15:34.082700 23813 external_mini_cluster.cc:1428] Setting key 9018349938ea7aa0d6792eb300daecfe
I20260810 18:15:34.083396 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.076162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.65:43189 (local address 127.23.65.126:40155)
0810 18:15:34.076305 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.076308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.077357 (+ 1049us) server_negotiation.cc:407] Connection header received
0810 18:15:34.078728 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.078731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.078789 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.078863 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:34.079815 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.080347 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.081484 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.081703 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.081793 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.082918 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:34.082922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:34.082925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:34.082939 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:34.082961 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:34.082969 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:34.083029 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.083180 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.083250 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:15:34.084293 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.076575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40155 (local address 127.23.65.65:43189)
0810 18:15:34.077210 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:34.077254 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:34.078521 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:34.079045 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:34.079052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:34.079137 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:34.079674 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.079689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.080471 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.080474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.081358 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.081368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.081853 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.081857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.082011 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.082643 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:15:34.082644 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:34.082685 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:15:34.082764 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:34.083039 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:34.083048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:34.083055 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:15:34.083164 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 18:15:34.083437 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260810 18:15:34.084649 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.66:0
--local_ip_for_outbound_sockets=127.23.65.66
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40155
--builtin_ntp_servers=127.23.65.84:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:15:34.087343 23856 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40155
I20260810 18:15:34.087672 23856 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:34.088289 23856 heartbeater.cc:511] Master 127.23.65.126:40155 requested a full tablet report, sending...
I20260810 18:15:34.090087 23533 ts_manager.cc:194] Registered new tserver with Master: ad70b297567c4357a45aa39cda7d7c3f (127.23.65.65:38891)
I20260810 18:15:34.091796 23533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.65:43189
W20260810 18:15:34.199064 23859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:34.199354 23859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:34.199398 23859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:34.203457 23859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:34.203517 23859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:34.203605 23859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.66
I20260810 18:15:34.208325 23859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.65.66
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.66
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:34.209438 23859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:34.210883 23859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:34.220358 23871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:34.220325 23872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:34.220325 23875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:34.220595 23859 server_base.cc:1034] running on GCE node
I20260810 18:15:34.221047 23859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:34.221681 23859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:34.222841 23859 hybrid_clock.cc:648] HybridClock initialized: now 1786385734222824 us; error 27 us; skew 500 ppm
I20260810 18:15:34.224758 23859 webserver.cc:460] Webserver started at http://127.23.65.66:36617/ using document root <none> and password file <none>
I20260810 18:15:34.225312 23859 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:34.225386 23859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:34.225593 23859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:34.227697 23859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/data/instance:
uuid: "2868c119aac640fd83394d5e3ac58cb7"
format_stamp: "Formatted at 2026-08-10 18:15:34 on dist-test-slave-jpj1"
server_key: "75694a46c2972a79f79fb325b7700f6b"
server_key_iv: "6fcd37f7bf2a8bb08c45d24fd2194a5b"
server_key_version: "encryptionkey@0"
I20260810 18:15:34.228251 23859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance:
uuid: "2868c119aac640fd83394d5e3ac58cb7"
format_stamp: "Formatted at 2026-08-10 18:15:34 on dist-test-slave-jpj1"
server_key: "75694a46c2972a79f79fb325b7700f6b"
server_key_iv: "6fcd37f7bf2a8bb08c45d24fd2194a5b"
server_key_version: "encryptionkey@0"
I20260810 18:15:34.231914 23859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:15:34.234308 23883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:34.235487 23859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:34.235618 23859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/wal
uuid: "2868c119aac640fd83394d5e3ac58cb7"
format_stamp: "Formatted at 2026-08-10 18:15:34 on dist-test-slave-jpj1"
server_key: "75694a46c2972a79f79fb325b7700f6b"
server_key_iv: "6fcd37f7bf2a8bb08c45d24fd2194a5b"
server_key_version: "encryptionkey@0"
I20260810 18:15:34.235714 23859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:34.258258 23859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:34.258975 23859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:34.259153 23859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:34.259814 23859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:34.260792 23859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:34.260841 23859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:34.260886 23859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:34.260915 23859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:34.269183 23859 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.66:44425
I20260810 18:15:34.269244 24071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.66:44425 every 8 connection(s)
I20260810 18:15:34.270078 23859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/data/info.pb
I20260810 18:15:34.272405 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 23859
I20260810 18:15:34.272522 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/wal/instance
I20260810 18:15:34.272807 23813 external_mini_cluster.cc:1428] Setting key 5f43606ce8bd0053ddb5990f9d5a2541
I20260810 18:15:34.274740 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.65.67:0
--local_ip_for_outbound_sockets=127.23.65.67
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40155
--builtin_ntp_servers=127.23.65.84:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:15:34.280169 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.271867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.66:58465 (local address 127.23.65.126:40155)
0810 18:15:34.271994 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.271998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.273198 (+ 1200us) server_negotiation.cc:407] Connection header received
0810 18:15:34.274665 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.274668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.274721 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.274818 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:34.275814 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.276534 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.278167 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.278383 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.278452 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.279719 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:34.279723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:34.279725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:34.279737 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:34.279763 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:34.279771 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:34.279827 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.280000 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.280055 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:15:34.281378 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.272249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40155 (local address 127.23.65.66:58465)
0810 18:15:34.273035 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:34.273089 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:34.274456 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:34.275016 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:34.275023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:34.275112 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:34.275678 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.275694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.276690 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.276695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.278048 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.278061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.278529 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.278532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.278717 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.279382 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:15:34.279385 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:34.279447 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:15:34.279529 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:34.279853 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:34.279862 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:34.279868 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:15:34.280001 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 18:15:34.280312 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":129,"threads_started":1}
I20260810 18:15:34.282457 24072 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40155
I20260810 18:15:34.282680 24072 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:34.283174 24072 heartbeater.cc:511] Master 127.23.65.126:40155 requested a full tablet report, sending...
I20260810 18:15:34.284154 23533 ts_manager.cc:194] Registered new tserver with Master: 2868c119aac640fd83394d5e3ac58cb7 (127.23.65.66:44425)
I20260810 18:15:34.284701 23533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.66:58465
W20260810 18:15:34.389266 24078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:34.389509 24078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:34.389547 24078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:34.393527 24078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:34.393589 24078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:34.393656 24078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.65.67
I20260810 18:15:34.398268 24078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.65.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.65.67
--webserver_port=0
--tserver_master_addrs=127.23.65.126:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.65.67
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:34.399266 24078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:34.400624 24078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:34.409668 24085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:34.409668 24084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:34.409773 24078 server_base.cc:1034] running on GCE node
W20260810 18:15:34.409703 24089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:34.410342 24078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:34.410948 24078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:34.412106 24078 hybrid_clock.cc:648] HybridClock initialized: now 1786385734412089 us; error 27 us; skew 500 ppm
I20260810 18:15:34.413834 24078 webserver.cc:460] Webserver started at http://127.23.65.67:33865/ using document root <none> and password file <none>
I20260810 18:15:34.414389 24078 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:34.414458 24078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:34.414660 24078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:34.416725 24078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/data/instance:
uuid: "19e97a129ee04f71b9b0ade15058a1e5"
format_stamp: "Formatted at 2026-08-10 18:15:34 on dist-test-slave-jpj1"
server_key: "a8ab59a74543f9bcaf05108f4b6638d5"
server_key_iv: "b014c8cd93131d21941a1f61b0557889"
server_key_version: "encryptionkey@0"
I20260810 18:15:34.417322 24078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance:
uuid: "19e97a129ee04f71b9b0ade15058a1e5"
format_stamp: "Formatted at 2026-08-10 18:15:34 on dist-test-slave-jpj1"
server_key: "a8ab59a74543f9bcaf05108f4b6638d5"
server_key_iv: "b014c8cd93131d21941a1f61b0557889"
server_key_version: "encryptionkey@0"
I20260810 18:15:34.420907 24078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:34.423182 24102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:34.424379 24078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:34.424490 24078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/wal
uuid: "19e97a129ee04f71b9b0ade15058a1e5"
format_stamp: "Formatted at 2026-08-10 18:15:34 on dist-test-slave-jpj1"
server_key: "a8ab59a74543f9bcaf05108f4b6638d5"
server_key_iv: "b014c8cd93131d21941a1f61b0557889"
server_key_version: "encryptionkey@0"
I20260810 18:15:34.424588 24078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:34.449477 24078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:34.450188 24078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:34.450368 24078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:34.450974 24078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:15:34.451947 24078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:15:34.451997 24078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:34.452054 24078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:15:34.452082 24078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:34.460424 24078 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.67:46017
I20260810 18:15:34.460460 24276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.67:46017 every 8 connection(s)
I20260810 18:15:34.461303 24078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/data/info.pb
I20260810 18:15:34.462257 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24078
I20260810 18:15:34.462361 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/wal/instance
I20260810 18:15:34.462642 23813 external_mini_cluster.cc:1428] Setting key 8281738d6f69d396852f3aa5614c12ff
I20260810 18:15:34.470433 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.463207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:37969 (local address 127.23.65.126:40155)
0810 18:15:34.463353 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.463358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.464527 (+ 1169us) server_negotiation.cc:407] Connection header received
0810 18:15:34.465930 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.465933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.465989 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.466063 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:34.466951 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.467469 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.468604 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.468824 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.468911 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.469969 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:34.469973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:34.469974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:34.469986 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:34.470005 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:34.470013 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:34.470069 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.470215 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.470288 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:15:34.471210 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.463693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:40155 (local address 127.23.65.67:37969)
0810 18:15:34.464396 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:34.464437 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:34.465721 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:34.466237 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:34.466245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:34.466329 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:15:34.466820 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.466832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.467603 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.467607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.468481 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.468491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.468965 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.468969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.469136 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.469711 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:15:34.469712 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:15:34.469759 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:15:34.469822 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:15:34.470080 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:15:34.470090 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:15:34.470096 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:15:34.470200 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 18:15:34.470464 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":137,"threads_started":1}
I20260810 18:15:34.472220 24277 heartbeater.cc:347] Connected to a master server at 127.23.65.126:40155
I20260810 18:15:34.472446 24277 heartbeater.cc:464] Registering TS with master...
I20260810 18:15:34.472921 24277 heartbeater.cc:511] Master 127.23.65.126:40155 requested a full tablet report, sending...
I20260810 18:15:34.473871 23533 ts_manager.cc:194] Registered new tserver with Master: 19e97a129ee04f71b9b0ade15058a1e5 (127.23.65.67:46017)
I20260810 18:15:34.474406 23533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.65.67:37969
I20260810 18:15:34.476514 23813 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:15:34.481417 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.478331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.23.65.126:40155)
0810 18:15:34.478442 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.478445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.478641 (+ 196us) server_negotiation.cc:407] Connection header received
0810 18:15:34.478730 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.478732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.478770 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.478828 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:34.479033 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.479504 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.480252 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.480445 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.480509 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.481112 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:34.481115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:34.481117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:34.481128 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:34.481145 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:34.481152 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:34.481209 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.481273 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.481320 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:15:34.483872 23533 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:15:34.486713 23533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:15:34.500737 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.495520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.23.65.65:38891)
0810 18:15:34.496065 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.496073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.496282 (+ 209us) server_negotiation.cc:407] Connection header received
0810 18:15:34.496392 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.496398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.496542 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.496671 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:34.497038 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.498207 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.499397 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.500041 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.500141 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.500339 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.500485 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.500583 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":130,"threads_started":1}
I20260810 18:15:34.500726 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.494722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.67:46017 (local address 127.0.0.1:38304)
0810 18:15:34.495475 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:34.495521 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:34.495674 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:34.496855 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:34.496858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:34.496877 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:34.496980 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.496988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.498029 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.498032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.499196 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.499206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.500240 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.500255 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.500356 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.500382 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:15:34.500427 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 18:15:34.500559 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":41,"thread_start_us":60,"threads_started":1}
I20260810 18:15:34.500730 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.495079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:38891 (local address 127.0.0.1:48410)
0810 18:15:34.496176 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:34.496188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:34.496281 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:34.496815 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:34.496818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:34.496834 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:34.496923 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.496929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.498341 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.498344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.499265 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.499276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.500162 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.500165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.500292 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.500343 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 18:15:34.500445 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 18:15:34.500571 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":30,"thread_start_us":99,"threads_started":1}
I20260810 18:15:34.501525 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.494872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.23.65.67:46017)
0810 18:15:34.496430 (+ 1558us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.496444 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.496464 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:15:34.496529 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.496534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.496689 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.496873 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:34.497084 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.497887 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.499320 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.500109 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.500948 (+ 839us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.501209 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.501273 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.501365 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1465,"thread_start_us":115,"threads_started":1}
I20260810 18:15:34.502017 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.494815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:44425 (local address 127.0.0.1:55502)
0810 18:15:34.496090 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:34.496100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:34.496189 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:34.496672 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:34.496677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:34.496698 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:34.496832 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.496839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.498341 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.498344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.500305 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.500313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.501512 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.501516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.501773 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.501798 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:15:34.501849 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 18:15:34.501915 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"mutex_wait_us":799,"thread_start_us":100,"threads_started":1}
I20260810 18:15:34.502730 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.495491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.23.65.66:44425)
0810 18:15:34.495971 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.495983 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.496192 (+ 209us) server_negotiation.cc:407] Connection header received
0810 18:15:34.496308 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.496314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.496512 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.496662 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:34.496969 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.498206 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.500429 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.501378 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.502268 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.502441 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.502489 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.502573 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":181,"threads_started":1}
I20260810 18:15:34.503883 23736 tablet_service.cc:1467] Processing CreateTablet for tablet 0957143716a34bb580682c3cb8f128c6 (DEFAULT_TABLE table=test-table [id=10b0414063b74662b3886038a483ef5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:34.504153 24173 tablet_service.cc:1467] Processing CreateTablet for tablet 0957143716a34bb580682c3cb8f128c6 (DEFAULT_TABLE table=test-table [id=10b0414063b74662b3886038a483ef5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:34.505040 24173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0957143716a34bb580682c3cb8f128c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:34.505046 23736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0957143716a34bb580682c3cb8f128c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:34.505296 23960 tablet_service.cc:1467] Processing CreateTablet for tablet 0957143716a34bb580682c3cb8f128c6 (DEFAULT_TABLE table=test-table [id=10b0414063b74662b3886038a483ef5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:15:34.506227 23960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0957143716a34bb580682c3cb8f128c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:34.511067 24311 tablet_bootstrap.cc:492] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: Bootstrap starting.
I20260810 18:15:34.512629 24312 tablet_bootstrap.cc:492] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f: Bootstrap starting.
I20260810 18:15:34.513211 24316 tablet_bootstrap.cc:492] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: Bootstrap starting.
I20260810 18:15:34.513273 24311 tablet_bootstrap.cc:654] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:34.514122 24311 log.cc:824] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:34.514580 24312 tablet_bootstrap.cc:654] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:34.515192 24316 tablet_bootstrap.cc:654] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:34.515303 24312 log.cc:824] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:34.516028 24316 log.cc:824] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:34.516458 24311 tablet_bootstrap.cc:492] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: No bootstrap required, opened a new log
I20260810 18:15:34.516642 24311 ts_tablet_manager.cc:1397] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:15:34.517196 24312 tablet_bootstrap.cc:492] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f: No bootstrap required, opened a new log
I20260810 18:15:34.517378 24312 ts_tablet_manager.cc:1397] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:15:34.517966 24316 tablet_bootstrap.cc:492] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: No bootstrap required, opened a new log
I20260810 18:15:34.518146 24316 ts_tablet_manager.cc:1397] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:15:34.520514 24312 raft_consensus.cc:348] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } }
I20260810 18:15:34.520726 24312 raft_consensus.cc:374] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:34.520779 24312 raft_consensus.cc:729] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad70b297567c4357a45aa39cda7d7c3f, State: Initialized, Role: FOLLOWER
I20260810 18:15:34.520817 24311 raft_consensus.cc:348] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } }
I20260810 18:15:34.521061 24311 raft_consensus.cc:374] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:34.521116 24311 raft_consensus.cc:729] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e97a129ee04f71b9b0ade15058a1e5, State: Initialized, Role: FOLLOWER
I20260810 18:15:34.521373 24312 consensus_queue.cc:260] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } }
I20260810 18:15:34.521602 24311 consensus_queue.cc:260] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } }
I20260810 18:15:34.522110 23856 heartbeater.cc:503] Master 127.23.65.126:40155 was elected leader, sending a full tablet report...
I20260810 18:15:34.522572 24311 ts_tablet_manager.cc:1428] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:15:34.522444 24316 raft_consensus.cc:348] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } }
I20260810 18:15:34.522675 24316 raft_consensus.cc:374] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:34.522716 24316 raft_consensus.cc:729] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2868c119aac640fd83394d5e3ac58cb7, State: Initialized, Role: FOLLOWER
I20260810 18:15:34.522905 24277 heartbeater.cc:503] Master 127.23.65.126:40155 was elected leader, sending a full tablet report...
I20260810 18:15:34.523144 24316 consensus_queue.cc:260] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } }
I20260810 18:15:34.523269 24312 ts_tablet_manager.cc:1428] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:15:34.524106 24316 ts_tablet_manager.cc:1428] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:15:34.524274 24072 heartbeater.cc:503] Master 127.23.65.126:40155 was elected leader, sending a full tablet report...
W20260810 18:15:34.575515 23857 tablet.cc:2369] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:15:34.712275 24280 tablet.cc:2369] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:15:34.770550 24323 raft_consensus.cc:482] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:15:34.770803 24323 raft_consensus.cc:504] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } }
W20260810 18:15:34.771183 24073 tablet.cc:2369] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:15:34.772207 24323 leader_election.cc:288] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2868c119aac640fd83394d5e3ac58cb7 (127.23.65.66:44425), ad70b297567c4357a45aa39cda7d7c3f (127.23.65.65:38891)
I20260810 18:15:34.776688 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.772549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:34109 (local address 127.23.65.66:44425)
0810 18:15:34.772673 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.772678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.772695 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:15:34.772858 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.772860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.772906 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.772971 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:34.773279 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.774025 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.775456 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.776070 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.776197 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.776253 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.776487 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.776554 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:15:34.776690 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.772602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.65.67:56719 (local address 127.23.65.65:38891)
0810 18:15:34.772732 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.772736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.773111 (+ 375us) server_negotiation.cc:407] Connection header received
0810 18:15:34.773206 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.773209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.773268 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.773342 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:15:34.773581 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.774262 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.775586 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.776162 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.776271 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.776313 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.776485 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.776554 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:15:34.776742 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.772472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.66:44425 (local address 127.23.65.67:34109)
0810 18:15:34.772623 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:34.772640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:34.772740 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:34.773011 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:34.773015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:34.773037 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:34.773169 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.773176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.774137 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.774139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.775333 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.775344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.776214 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.776218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.776330 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.776383 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 18:15:34.776468 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:15:34.776568 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":5}
I20260810 18:15:34.776921 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.772482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:38891 (local address 127.23.65.67:56719)
0810 18:15:34.773016 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:34.773030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:15:34.773115 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:15:34.773366 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:15:34.773368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:15:34.773384 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:15:34.773484 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.773488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.774357 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.774361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.775459 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:15:34.775468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.776272 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.776275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:15:34.776385 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.776403 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:15:34.776468 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:15:34.776568 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":186,"thread_start_us":145,"threads_started":1}
I20260810 18:15:34.777504 23770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0957143716a34bb580682c3cb8f128c6" candidate_uuid: "19e97a129ee04f71b9b0ade15058a1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad70b297567c4357a45aa39cda7d7c3f" is_pre_election: true
I20260810 18:15:34.777501 23999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0957143716a34bb580682c3cb8f128c6" candidate_uuid: "19e97a129ee04f71b9b0ade15058a1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2868c119aac640fd83394d5e3ac58cb7" is_pre_election: true
I20260810 18:15:34.777791 23999 raft_consensus.cc:2454] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e97a129ee04f71b9b0ade15058a1e5 in term 0.
I20260810 18:15:34.777807 23770 raft_consensus.cc:2454] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e97a129ee04f71b9b0ade15058a1e5 in term 0.
I20260810 18:15:34.778298 24103 leader_election.cc:302] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19e97a129ee04f71b9b0ade15058a1e5, 2868c119aac640fd83394d5e3ac58cb7, ad70b297567c4357a45aa39cda7d7c3f; no voters:
I20260810 18:15:34.778553 24323 raft_consensus.cc:2781] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:15:34.778668 24323 raft_consensus.cc:482] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:15:34.778705 24323 raft_consensus.cc:3037] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:34.779815 24323 raft_consensus.cc:504] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } }
I20260810 18:15:34.780126 24323 leader_election.cc:288] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [CANDIDATE]: Term 1 election: Requested vote from peers 2868c119aac640fd83394d5e3ac58cb7 (127.23.65.66:44425), ad70b297567c4357a45aa39cda7d7c3f (127.23.65.65:38891)
I20260810 18:15:34.780478 23770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0957143716a34bb580682c3cb8f128c6" candidate_uuid: "19e97a129ee04f71b9b0ade15058a1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad70b297567c4357a45aa39cda7d7c3f"
I20260810 18:15:34.780479 23999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0957143716a34bb580682c3cb8f128c6" candidate_uuid: "19e97a129ee04f71b9b0ade15058a1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2868c119aac640fd83394d5e3ac58cb7"
I20260810 18:15:34.780587 23770 raft_consensus.cc:3037] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:34.780594 23999 raft_consensus.cc:3037] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:34.781780 23770 raft_consensus.cc:2454] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e97a129ee04f71b9b0ade15058a1e5 in term 1.
I20260810 18:15:34.781855 23999 raft_consensus.cc:2454] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e97a129ee04f71b9b0ade15058a1e5 in term 1.
I20260810 18:15:34.782088 24103 leader_election.cc:302] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e97a129ee04f71b9b0ade15058a1e5, ad70b297567c4357a45aa39cda7d7c3f; no voters:
I20260810 18:15:34.782277 24323 raft_consensus.cc:2781] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:34.782498 24323 raft_consensus.cc:686] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 1 LEADER]: Becoming Leader. State: Replica: 19e97a129ee04f71b9b0ade15058a1e5, State: Running, Role: LEADER
I20260810 18:15:34.782833 24323 consensus_queue.cc:237] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } }
I20260810 18:15:34.785810 23531 catalog_manager.cc:5495] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19e97a129ee04f71b9b0ade15058a1e5 (127.23.65.67). New cstate: current_term: 1 leader_uuid: "19e97a129ee04f71b9b0ade15058a1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e97a129ee04f71b9b0ade15058a1e5" member_type: VOTER last_known_addr { host: "127.23.65.67" port: 46017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260810 18:15:34.860280 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.856968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.23.65.67:46017)
0810 18:15:34.857098 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:34.857102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:34.857116 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:15:34.857197 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:34.857199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:34.857253 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:34.857345 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:15:34.857530 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.858019 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.858896 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:34.859131 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:34.859204 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:34.859448 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:15:34.859909 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:15:34.860001 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:34.860070 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 18:15:34.860116 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:15:34.866016 23999 raft_consensus.cc:1261] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [term 1 FOLLOWER]: Refusing update from remote peer 19e97a129ee04f71b9b0ade15058a1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:34.866017 23770 raft_consensus.cc:1261] T 0957143716a34bb580682c3cb8f128c6 P ad70b297567c4357a45aa39cda7d7c3f [term 1 FOLLOWER]: Refusing update from remote peer 19e97a129ee04f71b9b0ade15058a1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:15:34.866714 24323 consensus_queue.cc:1035] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2868c119aac640fd83394d5e3ac58cb7" member_type: VOTER last_known_addr { host: "127.23.65.66" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:34.866927 24328 consensus_queue.cc:1035] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad70b297567c4357a45aa39cda7d7c3f" member_type: VOTER last_known_addr { host: "127.23.65.65" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:15:34.873636 24332 mvcc.cc:205] Tried to move back new op lower bound from 7317035970004135936 to 7317035969672953856. Current Snapshot: MvccSnapshot[applied={T|T < 7317035970004135936}]
I20260810 18:15:34.874899 24334 mvcc.cc:205] Tried to move back new op lower bound from 7317035970004135936 to 7317035969672953856. Current Snapshot: MvccSnapshot[applied={T|T < 7317035970004135936}]
I20260810 18:15:34.882541 23531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:34.882717 23531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:15:34.885219 23531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77e803b2400149029a73e3eca1ae938f: Sending DeleteTablet for 3 replicas of tablet 0957143716a34bb580682c3cb8f128c6
I20260810 18:15:34.885974 24173 tablet_service.cc:1514] Processing DeleteTablet for tablet 0957143716a34bb580682c3cb8f128c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:34 UTC) from {username='slave'} at 127.0.0.1:38304
I20260810 18:15:34.885968 23960 tablet_service.cc:1514] Processing DeleteTablet for tablet 0957143716a34bb580682c3cb8f128c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:34 UTC) from {username='slave'} at 127.0.0.1:55502
I20260810 18:15:34.886119 23736 tablet_service.cc:1514] Processing DeleteTablet for tablet 0957143716a34bb580682c3cb8f128c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:15:34 UTC) from {username='slave'} at 127.0.0.1:48410
I20260810 18:15:34.886483 24350 tablet_replica.cc:330] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: stopping tablet replica
I20260810 18:15:34.886816 24350 raft_consensus.cc:2229] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:15:34.887281 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 23628
I20260810 18:15:34.897953 24353 tablet_replica.cc:330] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: stopping tablet replica
I20260810 18:15:34.898274 24353 raft_consensus.cc:2229] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:15:34.898615 24353 raft_consensus.cc:2258] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:34.898833 24350 raft_consensus.cc:2258] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:15:34.901437 24353 ts_tablet_manager.cc:1905] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:34.901639 24350 ts_tablet_manager.cc:1905] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:15:34.904372 24353 ts_tablet_manager.cc:1918] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:34.904372 24350 ts_tablet_manager.cc:1918] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:15:34.904489 24353 log.cc:1196] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-2/wal/wals/0957143716a34bb580682c3cb8f128c6
I20260810 18:15:34.904489 24350 log.cc:1196] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: Deleting WAL directory at /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385478759079-23813-0/minicluster-data/ts-1/wal/wals/0957143716a34bb580682c3cb8f128c6
I20260810 18:15:34.904824 24353 ts_tablet_manager.cc:1939] T 0957143716a34bb580682c3cb8f128c6 P 19e97a129ee04f71b9b0ade15058a1e5: Deleting consensus metadata
I20260810 18:15:34.904824 24350 ts_tablet_manager.cc:1939] T 0957143716a34bb580682c3cb8f128c6 P 2868c119aac640fd83394d5e3ac58cb7: Deleting consensus metadata
I20260810 18:15:34.906097 23513 catalog_manager.cc:4841] TS 19e97a129ee04f71b9b0ade15058a1e5 (127.23.65.67:46017): tablet 0957143716a34bb580682c3cb8f128c6 (table test-table [id=10b0414063b74662b3886038a483ef5e]) successfully deleted
I20260810 18:15:34.906097 23516 catalog_manager.cc:4841] TS 2868c119aac640fd83394d5e3ac58cb7 (127.23.65.66:44425): tablet 0957143716a34bb580682c3cb8f128c6 (table test-table [id=10b0414063b74662b3886038a483ef5e]) successfully deleted
W20260810 18:15:34.908999 23507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.65.65:38891 (error 108)
I20260810 18:15:34.909207 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 23859
I20260810 18:15:34.910185 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:34.909888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.65:38891 (local address 127.0.0.1:48418)
0810 18:15:34.910044 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 18:15:34.910102 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260810 18:15:34.910326 23507 catalog_manager.cc:4584] TS ad70b297567c4357a45aa39cda7d7c3f (127.23.65.65:38891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0957143716a34bb580682c3cb8f128c6: Network error: Client connection negotiation failed: client connection to 127.23.65.65:38891: connect: Connection refused (error 111)
I20260810 18:15:34.923655 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24078
I20260810 18:15:34.935556 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 23485
2026-08-10T18:15:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1319 ms)
[----------] 6 tests from EncryptionPolicyTest (11369 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:34 dist-test-slave-jpj1 krb5kdc[24359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:34 dist-test-slave-jpj1 krb5kdc[24359](info): set up 2 sockets
Aug 10 18:15:34 dist-test-slave-jpj1 krb5kdc[24359](info): commencing operation
krb5kdc: starting...
W20260810 18:15:36.094024 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:36 dist-test-slave-jpj1 krb5kdc[24359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:36.218425 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:36.333891 24379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:36.334167 24379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:36.334215 24379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:36.338239 24379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:36.338303 24379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:36.338321 24379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:36.338336 24379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:36.338351 24379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:36.343662 24379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:36.344740 24379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:36.345804 24379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:36.354472 24392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:36.354506 24390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:36.354506 24389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:36.354839 24379 server_base.cc:1034] running on GCE node
I20260810 18:15:36.355424 24379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:36.356557 24379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:36.357720 24379 hybrid_clock.cc:648] HybridClock initialized: now 1786385736357708 us; error 28 us; skew 500 ppm
Aug 10 18:15:36 dist-test-slave-jpj1 krb5kdc[24359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:36.368451 24379 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:36.369441 24379 webserver.cc:460] Webserver started at http://127.0.0.1:34047/ using document root <none> and password file <none>
I20260810 18:15:36.370019 24379 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:36.370082 24379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:36.370254 24379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:36.372385 24379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "3a50bad7afc34996bcc6914c362ce999"
format_stamp: "Formatted at 2026-08-10 18:15:36 on dist-test-slave-jpj1"
server_key: "69c953d749441a83f700944f25daa73f"
server_key_iv: "989e97a9f5acefd02a7c239105415783"
server_key_version: "encryptionkey@0"
I20260810 18:15:36.372922 24379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "3a50bad7afc34996bcc6914c362ce999"
format_stamp: "Formatted at 2026-08-10 18:15:36 on dist-test-slave-jpj1"
server_key: "69c953d749441a83f700944f25daa73f"
server_key_iv: "989e97a9f5acefd02a7c239105415783"
server_key_version: "encryptionkey@0"
I20260810 18:15:36.376808 24379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 18:15:36.379304 24404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:36.380612 24379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:36.380720 24379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "3a50bad7afc34996bcc6914c362ce999"
format_stamp: "Formatted at 2026-08-10 18:15:36 on dist-test-slave-jpj1"
server_key: "69c953d749441a83f700944f25daa73f"
server_key_iv: "989e97a9f5acefd02a7c239105415783"
server_key_version: "encryptionkey@0"
I20260810 18:15:36.380836 24379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:36.391121 24379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:36.394202 24379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:36.394356 24379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:36.401448 24379 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33259
I20260810 18:15:36.401463 24494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33259 every 8 connection(s)
I20260810 18:15:36.402419 24379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:36.405195 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24379
I20260810 18:15:36.405287 24498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:36.405325 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:36.405586 23813 external_mini_cluster.cc:1428] Setting key 43e379fd636e30a9dd2abe650ff08d15
I20260810 18:15:36.412403 24498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999: Bootstrap starting.
Aug 10 18:15:36 dist-test-slave-jpj1 krb5kdc[24359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:15:36.415184 24498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:36.415987 24498 log.cc:824] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:36.418366 24498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999: No bootstrap required, opened a new log
I20260810 18:15:36.420693 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:36.407018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44009 (local address 127.0.0.1:33259)
0810 18:15:36.407430 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:36.407444 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:36.407479 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:15:36.408363 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:36.408381 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:36.408723 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:36.409026 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:36.410014 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:36.410981 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:36.412158 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:36.412640 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:36.412763 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:36.415471 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:36.415476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:36.415491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:36.415519 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:36.417436 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:36.417995 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:36.418001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:36.418007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:36.418166 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:36.418463 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:36.418466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:36.418468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:36.418981 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:36.419210 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:36.419454 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 18:15:36.419712 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260810 18:15:36.422493 24498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a50bad7afc34996bcc6914c362ce999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260810 18:15:36.422748 24498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:36.422819 24498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a50bad7afc34996bcc6914c362ce999, State: Initialized, Role: FOLLOWER
I20260810 18:15:36.423259 24498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a50bad7afc34996bcc6914c362ce999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260810 18:15:36.423399 24498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:36.423462 24498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:36.423588 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:36.423591 24498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:36.424487 24498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a50bad7afc34996bcc6914c362ce999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260810 18:15:36.424870 24498 leader_election.cc:302] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a50bad7afc34996bcc6914c362ce999; no voters:
I20260810 18:15:36.425173 24498 leader_election.cc:288] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:36.425287 24512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:36.425472 24512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [term 1 LEADER]: Becoming Leader. State: Replica: 3a50bad7afc34996bcc6914c362ce999, State: Running, Role: LEADER
I20260810 18:15:36.425750 24512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a50bad7afc34996bcc6914c362ce999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260810 18:15:36.426469 24498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:36.427459 24517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a50bad7afc34996bcc6914c362ce999. Latest consensus state: current_term: 1 leader_uuid: "3a50bad7afc34996bcc6914c362ce999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a50bad7afc34996bcc6914c362ce999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } } }
I20260810 18:15:36.427606 24517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:36.428050 24528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:36.428051 24516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a50bad7afc34996bcc6914c362ce999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a50bad7afc34996bcc6914c362ce999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } } }
I20260810 18:15:36.428182 24516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:36.431213 24528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:36.435518 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:36.425488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57079 (local address 127.0.0.1:33259)
0810 18:15:36.425596 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:36.425599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:36.425841 (+ 242us) server_negotiation.cc:407] Connection header received
0810 18:15:36.426015 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:36.426018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:36.426059 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:36.426103 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:36.426869 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:36.427664 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:36.428741 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:36.428979 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:36.429069 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:36.431860 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:36.431864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:36.431867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:36.431894 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:36.433563 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:36.434112 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:36.434115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:36.434117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:36.434211 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:36.434564 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:36.434568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:36.434570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:36.434801 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:36.434952 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:36.435258 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 18:15:36.435361 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:15:36.437117 24528 catalog_manager.cc:1347] Generated new cluster ID: d7577ecc51af450f95ad92f815c4e799
I20260810 18:15:36.437179 24528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:36.455267 24528 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:36.456594 24528 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:36.465953 24528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a50bad7afc34996bcc6914c362ce999: Generated new TSK 0
I20260810 18:15:36.466811 24528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:36.472476 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24379
W20260810 18:15:36.484958 24356 connection.cc:535] client connection to 127.0.0.1:33259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33259): Connection reset by peer (error 104)
2026-08-10T18:15:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1536 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:36 dist-test-slave-jpj1 krb5kdc[24555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:36 dist-test-slave-jpj1 krb5kdc[24555](info): set up 2 sockets
Aug 10 18:15:36 dist-test-slave-jpj1 krb5kdc[24555](info): commencing operation
krb5kdc: starting...
W20260810 18:15:37.639830 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:37 dist-test-slave-jpj1 krb5kdc[24555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:37.762100 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:37.872575 24581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:37.872843 24581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:37.872896 24581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:37.876823 24581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:37.876884 24581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:37.876909 24581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:37.876932 24581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:37.876960 24581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:37.882064 24581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:37.883057 24581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:37.884114 24581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:37.892318 24595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:37.892313 24599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:37.892345 24592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:37.892513 24581 server_base.cc:1034] running on GCE node
I20260810 18:15:37.893194 24581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:37.894253 24581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:37.895411 24581 hybrid_clock.cc:648] HybridClock initialized: now 1786385737895397 us; error 26 us; skew 500 ppm
Aug 10 18:15:37 dist-test-slave-jpj1 krb5kdc[24555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:37.898090 24581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:37.898970 24581 webserver.cc:460] Webserver started at http://127.0.0.1:43447/ using document root <none> and password file <none>
I20260810 18:15:37.899500 24581 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:37.899600 24581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:37.899806 24581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:37.901830 24581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "b5424d8aff7b462c8d1ee1cd801b7566"
format_stamp: "Formatted at 2026-08-10 18:15:37 on dist-test-slave-jpj1"
server_key: "f0d9d7848c8b7d407f8cd4aa101280f2"
server_key_iv: "953a20f7689522b53f12fa9fa677d655"
server_key_version: "encryptionkey@0"
I20260810 18:15:37.902379 24581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "b5424d8aff7b462c8d1ee1cd801b7566"
format_stamp: "Formatted at 2026-08-10 18:15:37 on dist-test-slave-jpj1"
server_key: "f0d9d7848c8b7d407f8cd4aa101280f2"
server_key_iv: "953a20f7689522b53f12fa9fa677d655"
server_key_version: "encryptionkey@0"
I20260810 18:15:37.906026 24581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:15:37.908298 24614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:37.909451 24581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:37.909564 24581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "b5424d8aff7b462c8d1ee1cd801b7566"
format_stamp: "Formatted at 2026-08-10 18:15:37 on dist-test-slave-jpj1"
server_key: "f0d9d7848c8b7d407f8cd4aa101280f2"
server_key_iv: "953a20f7689522b53f12fa9fa677d655"
server_key_version: "encryptionkey@0"
I20260810 18:15:37.909669 24581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:37.941025 24581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:37.944218 24581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:37.944409 24581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:37.951591 24581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38973
I20260810 18:15:37.951622 24686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38973 every 8 connection(s)
I20260810 18:15:37.952579 24581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:37.955627 24687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:37.958709 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24581
I20260810 18:15:37.958846 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:37.959106 23813 external_mini_cluster.cc:1428] Setting key daf3fdaea6a1576a55a6fe803a38aad8
I20260810 18:15:37.961740 24687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566: Bootstrap starting.
I20260810 18:15:37.964488 24687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:37.965269 24687 log.cc:824] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:37.967433 24687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566: No bootstrap required, opened a new log
I20260810 18:15:37.968977 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:37.960538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49829 (local address 127.0.0.1:38973)
0810 18:15:37.960982 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:37.961001 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:37.961045 (+ 44us) server_negotiation.cc:407] Connection header received
0810 18:15:37.961858 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:37.961882 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:37.962320 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:37.962676 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:37.963115 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:37.964626 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:37.965517 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:37.966076 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:37.966227 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:37.967025 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:37.967031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:37.967048 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:37.967566 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:37.967635 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:37.967666 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:37.967796 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:37.967930 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 18:15:37.968064 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260810 18:15:37.971652 24687 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5424d8aff7b462c8d1ee1cd801b7566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260810 18:15:37.971935 24687 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:37.971998 24687 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5424d8aff7b462c8d1ee1cd801b7566, State: Initialized, Role: FOLLOWER
I20260810 18:15:37.972143 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:37.972440 24687 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5424d8aff7b462c8d1ee1cd801b7566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260810 18:15:37.972568 24687 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:37.972630 24687 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:37.972728 24687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:37.973634 24687 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5424d8aff7b462c8d1ee1cd801b7566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260810 18:15:37.973991 24687 leader_election.cc:302] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5424d8aff7b462c8d1ee1cd801b7566; no voters:
I20260810 18:15:37.974327 24687 leader_election.cc:288] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:37.974411 24705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:37.974649 24705 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [term 1 LEADER]: Becoming Leader. State: Replica: b5424d8aff7b462c8d1ee1cd801b7566, State: Running, Role: LEADER
I20260810 18:15:37.975031 24705 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5424d8aff7b462c8d1ee1cd801b7566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260810 18:15:37.975584 24687 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:37.976253 24705 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5424d8aff7b462c8d1ee1cd801b7566. Latest consensus state: current_term: 1 leader_uuid: "b5424d8aff7b462c8d1ee1cd801b7566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5424d8aff7b462c8d1ee1cd801b7566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } } }
I20260810 18:15:37.976998 24705 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:37.976888 24707 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5424d8aff7b462c8d1ee1cd801b7566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5424d8aff7b462c8d1ee1cd801b7566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } } }
I20260810 18:15:37.977319 24707 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:37.979018 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:37.974104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35193 (local address 127.0.0.1:38973)
0810 18:15:37.974214 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:37.974218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:37.974419 (+ 201us) server_negotiation.cc:407] Connection header received
0810 18:15:37.974588 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:37.974591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:37.974637 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:37.974709 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:37.975109 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:37.975933 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:37.977048 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:37.977380 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:37.977455 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:37.978612 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:37.978616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:37.978618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:37.978632 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:37.978660 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:37.978673 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:37.978766 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:37.978854 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 18:15:37.978890 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 18:15:37.980836 24727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:15:37.980903 24727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:15:37.980976 24719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:37.981679 24719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:37.986490 24719 catalog_manager.cc:1347] Generated new cluster ID: 7df81f42c7174c7bb6c1c4862514b4e6
I20260810 18:15:37.986550 24719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:38.014097 24719 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:38.015444 24719 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:38.022197 24719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5424d8aff7b462c8d1ee1cd801b7566: Generated new TSK 0
I20260810 18:15:38.023072 24719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:38.030146 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24581
W20260810 18:15:38.048590 24551 connection.cc:535] client connection to 127.0.0.1:38973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38973): Connection reset by peer (error 104)
2026-08-10T18:15:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1563 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:38 dist-test-slave-jpj1 krb5kdc[24734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:38 dist-test-slave-jpj1 krb5kdc[24734](info): set up 2 sockets
Aug 10 18:15:38 dist-test-slave-jpj1 krb5kdc[24734](info): commencing operation
krb5kdc: starting...
W20260810 18:15:39.193841 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:39 dist-test-slave-jpj1 krb5kdc[24734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:39.315640 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:39.429075 24755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:39.429343 24755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:39.429409 24755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:39.433403 24755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:39.433475 24755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:39.433501 24755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:39.433523 24755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:39.433552 24755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:39.438866 24755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:39.440011 24755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:39.441044 24755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:39.449412 24768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:39.449405 24764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:39.449465 24755 server_base.cc:1034] running on GCE node
W20260810 18:15:39.449405 24766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:39.450215 24755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:39.451375 24755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:39.452533 24755 hybrid_clock.cc:648] HybridClock initialized: now 1786385739452511 us; error 53 us; skew 500 ppm
Aug 10 18:15:39 dist-test-slave-jpj1 krb5kdc[24734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:39.455137 24755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:39.456046 24755 webserver.cc:460] Webserver started at http://127.0.0.1:43877/ using document root <none> and password file <none>
I20260810 18:15:39.456569 24755 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:39.456635 24755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:39.456840 24755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:39.458829 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "87083cb5a8db4f97a12acd3c905bb686"
format_stamp: "Formatted at 2026-08-10 18:15:39 on dist-test-slave-jpj1"
server_key: "08b2fd5f9ab6cec18d63e86c83ef7b09"
server_key_iv: "051c7ac00c5f0960af356da5d1a84a1c"
server_key_version: "encryptionkey@0"
I20260810 18:15:39.459385 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "87083cb5a8db4f97a12acd3c905bb686"
format_stamp: "Formatted at 2026-08-10 18:15:39 on dist-test-slave-jpj1"
server_key: "08b2fd5f9ab6cec18d63e86c83ef7b09"
server_key_iv: "051c7ac00c5f0960af356da5d1a84a1c"
server_key_version: "encryptionkey@0"
I20260810 18:15:39.463099 24755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:15:39.465422 24784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:39.466576 24755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:39.466703 24755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "87083cb5a8db4f97a12acd3c905bb686"
format_stamp: "Formatted at 2026-08-10 18:15:39 on dist-test-slave-jpj1"
server_key: "08b2fd5f9ab6cec18d63e86c83ef7b09"
server_key_iv: "051c7ac00c5f0960af356da5d1a84a1c"
server_key_version: "encryptionkey@0"
I20260810 18:15:39.466796 24755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:39.471279 24755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:39.473932 24755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:39.474094 24755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:39.480732 24755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35009
I20260810 18:15:39.480800 24875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35009 every 8 connection(s)
I20260810 18:15:39.481707 24755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:39.484442 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:39.489795 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686: Bootstrap starting.
I20260810 18:15:39.491348 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24755
I20260810 18:15:39.491482 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:39.491786 23813 external_mini_cluster.cc:1428] Setting key 2298d775b09ce4eba749c246a9c55123
I20260810 18:15:39.492234 24876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:39.493100 24876 log.cc:824] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:39.495076 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686: No bootstrap required, opened a new log
I20260810 18:15:39.496627 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:39.492833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45783 (local address 127.0.0.1:35009)
0810 18:15:39.493244 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:39.493263 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:39.493301 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:15:39.494064 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:39.494084 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:39.494365 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:39.494637 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:39.495438 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:39.495442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:39.495453 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:39.495463 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:39.495516 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:39.495558 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:39.495660 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:39.495767 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 18:15:39.495913 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260810 18:15:39.499079 24876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87083cb5a8db4f97a12acd3c905bb686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35009 } }
I20260810 18:15:39.499316 24876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:39.499390 24876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87083cb5a8db4f97a12acd3c905bb686, State: Initialized, Role: FOLLOWER
I20260810 18:15:39.499418 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:39.499845 24876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87083cb5a8db4f97a12acd3c905bb686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35009 } }
I20260810 18:15:39.499979 24876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:39.500046 24876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:39.500142 24876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:39.501210 24876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87083cb5a8db4f97a12acd3c905bb686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35009 } }
I20260810 18:15:39.501567 24876 leader_election.cc:302] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87083cb5a8db4f97a12acd3c905bb686; no voters:
I20260810 18:15:39.501909 24876 leader_election.cc:288] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:39.501978 24895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:39.502187 24895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [term 1 LEADER]: Becoming Leader. State: Replica: 87083cb5a8db4f97a12acd3c905bb686, State: Running, Role: LEADER
I20260810 18:15:39.502526 24895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87083cb5a8db4f97a12acd3c905bb686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35009 } }
I20260810 18:15:39.503153 24876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:39.503185 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:39.501421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41711 (local address 127.0.0.1:35009)
0810 18:15:39.501543 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:39.501547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:39.501755 (+ 208us) server_negotiation.cc:407] Connection header received
0810 18:15:39.501900 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:39.501903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:39.501939 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:39.502019 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:39.502796 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:39.502807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:39.502810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:39.502822 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:39.502850 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:39.502862 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:39.502946 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:39.503031 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 18:15:39.503067 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:15:39.504287 24895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87083cb5a8db4f97a12acd3c905bb686. Latest consensus state: current_term: 1 leader_uuid: "87083cb5a8db4f97a12acd3c905bb686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87083cb5a8db4f97a12acd3c905bb686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35009 } } }
I20260810 18:15:39.504410 24895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:39.504717 24899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87083cb5a8db4f97a12acd3c905bb686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87083cb5a8db4f97a12acd3c905bb686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35009 } } }
I20260810 18:15:39.504810 24899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:39.504845 24910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:39.507939 24910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:39.513243 24910 catalog_manager.cc:1347] Generated new cluster ID: 14c106a84113421b9a971d21b5ef4300
I20260810 18:15:39.513314 24910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:39.555941 24910 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:39.556406 24910 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:39.564049 24910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87083cb5a8db4f97a12acd3c905bb686: Generated new TSK 0
I20260810 18:15:39.564939 24910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:39.573033 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24755
2026-08-10T18:15:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1540 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:39 dist-test-slave-jpj1 krb5kdc[24933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:39 dist-test-slave-jpj1 krb5kdc[24933](info): set up 2 sockets
Aug 10 18:15:39 dist-test-slave-jpj1 krb5kdc[24933](info): commencing operation
krb5kdc: starting...
W20260810 18:15:40.745731 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:40 dist-test-slave-jpj1 krb5kdc[24933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:40.866278 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:40.974575 24949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:40.974838 24949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:40.974896 24949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:40.978832 24949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:40.978891 24949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:40.978919 24949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:40.978941 24949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:40.978967 24949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:40.984133 24949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:40.985145 24949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:40.986160 24949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:15:40.994580 24949 server_base.cc:1034] running on GCE node
W20260810 18:15:40.994513 24960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:40.994534 24957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:40.994509 24958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:40.995396 24949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:40.996523 24949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:40.997685 24949 hybrid_clock.cc:648] HybridClock initialized: now 1786385740997672 us; error 26 us; skew 500 ppm
Aug 10 18:15:40 dist-test-slave-jpj1 krb5kdc[24933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:41.000530 24949 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:41.001464 24949 webserver.cc:460] Webserver started at http://127.0.0.1:33225/ using document root <none> and password file <none>
I20260810 18:15:41.002005 24949 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:41.002074 24949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:41.002264 24949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:41.004346 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "c6451ab6b9c944f296a361194c0dc21a"
format_stamp: "Formatted at 2026-08-10 18:15:40 on dist-test-slave-jpj1"
server_key: "1ca417f03e6c67049b045e698000ad4a"
server_key_iv: "762585c2b4726cb762be1bc224816f88"
server_key_version: "encryptionkey@0"
I20260810 18:15:41.004881 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "c6451ab6b9c944f296a361194c0dc21a"
format_stamp: "Formatted at 2026-08-10 18:15:40 on dist-test-slave-jpj1"
server_key: "1ca417f03e6c67049b045e698000ad4a"
server_key_iv: "762585c2b4726cb762be1bc224816f88"
server_key_version: "encryptionkey@0"
I20260810 18:15:41.008488 24949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:41.010794 24976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:41.012023 24949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:41.012140 24949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "c6451ab6b9c944f296a361194c0dc21a"
format_stamp: "Formatted at 2026-08-10 18:15:40 on dist-test-slave-jpj1"
server_key: "1ca417f03e6c67049b045e698000ad4a"
server_key_iv: "762585c2b4726cb762be1bc224816f88"
server_key_version: "encryptionkey@0"
I20260810 18:15:41.012234 24949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:41.036705 24949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:41.040009 24949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:41.040202 24949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:41.047843 25074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46163 every 8 connection(s)
I20260810 18:15:41.047880 24949 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46163
I20260810 18:15:41.048926 24949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:41.051801 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:41.052095 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 24949
I20260810 18:15:41.052228 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:41.052481 23813 external_mini_cluster.cc:1428] Setting key 368e3dda14464d2eb12e7443aa2a8760
I20260810 18:15:41.058791 25075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a: Bootstrap starting.
Aug 10 18:15:41 dist-test-slave-jpj1 krb5kdc[24933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:15:41.061619 25075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:41.062511 25075 log.cc:824] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:41.065253 25075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a: No bootstrap required, opened a new log
I20260810 18:15:41.067826 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:41.053831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42625 (local address 127.0.0.1:46163)
0810 18:15:41.054252 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:41.054268 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:41.054305 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:15:41.054978 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:41.054997 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:41.055350 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:41.055695 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:41.056599 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:41.057608 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:41.058792 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:41.059323 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:41.059446 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:41.062220 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:41.062227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:41.062253 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:41.062288 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:41.064363 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:41.064846 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:41.064852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:41.064858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:41.065004 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:41.065403 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:41.065407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:41.065409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:41.065957 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:41.066166 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:41.066417 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 18:15:41.066742 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
I20260810 18:15:41.069904 25075 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6451ab6b9c944f296a361194c0dc21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46163 } }
I20260810 18:15:41.070176 25075 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:41.070277 25075 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6451ab6b9c944f296a361194c0dc21a, State: Initialized, Role: FOLLOWER
I20260810 18:15:41.070775 25075 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6451ab6b9c944f296a361194c0dc21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46163 } }
I20260810 18:15:41.070950 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:41.071018 25075 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:41.071099 25075 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:41.071204 25075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:41.072293 25075 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6451ab6b9c944f296a361194c0dc21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46163 } }
I20260810 18:15:41.072695 25075 leader_election.cc:302] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6451ab6b9c944f296a361194c0dc21a; no voters:
I20260810 18:15:41.073053 25075 leader_election.cc:288] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:41.073169 25088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:41.073529 25088 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [term 1 LEADER]: Becoming Leader. State: Replica: c6451ab6b9c944f296a361194c0dc21a, State: Running, Role: LEADER
I20260810 18:15:41.073917 25088 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6451ab6b9c944f296a361194c0dc21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46163 } }
I20260810 18:15:41.074509 25075 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:41.075083 25088 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6451ab6b9c944f296a361194c0dc21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6451ab6b9c944f296a361194c0dc21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46163 } } }
I20260810 18:15:41.075237 25088 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:41.075456 25088 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6451ab6b9c944f296a361194c0dc21a. Latest consensus state: current_term: 1 leader_uuid: "c6451ab6b9c944f296a361194c0dc21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6451ab6b9c944f296a361194c0dc21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46163 } } }
I20260810 18:15:41.075605 25088 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a [sys.catalog]: This master's current role is: LEADER
W20260810 18:15:41.080016 25114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:15:41.080097 25114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:15:41.080183 25101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:41.081056 25101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:41.082365 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:41.073443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53321 (local address 127.0.0.1:46163)
0810 18:15:41.073570 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:41.073574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:41.073589 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:15:41.073627 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:41.073630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:41.073676 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:41.073760 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:41.074606 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:41.075394 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:41.076447 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:41.076780 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:41.076877 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:41.078738 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:41.078741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:41.078744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:41.078770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:41.080435 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:41.080933 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:41.080936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:41.080938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:41.081037 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:41.081353 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:41.081356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:41.081358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:41.081616 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:41.081765 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:41.082038 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 18:15:41.082176 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:15:41.086808 25101 catalog_manager.cc:1347] Generated new cluster ID: cea180ea9b3c46e2a707a093162b80f4
I20260810 18:15:41.086880 25101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:41.104803 25101 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:41.105705 25101 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:41.111133 25101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6451ab6b9c944f296a361194c0dc21a: Generated new TSK 0
I20260810 18:15:41.112002 25101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:41.117507 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 24949
2026-08-10T18:15:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1539 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:41 dist-test-slave-jpj1 krb5kdc[25121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:41 dist-test-slave-jpj1 krb5kdc[25121](info): set up 2 sockets
Aug 10 18:15:41 dist-test-slave-jpj1 krb5kdc[25121](info): commencing operation
krb5kdc: starting...
W20260810 18:15:42.275730 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:42 dist-test-slave-jpj1 krb5kdc[25121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:42.396428 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:42.509647 25143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:42.509908 25143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:42.509948 25143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:42.513897 25143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:42.513947 25143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:42.513965 25143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:42.513980 25143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:42.513998 25143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:42.519104 25143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:42.520141 25143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:42.521123 25143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:42.529333 25154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:42.529395 25152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:42.529402 25160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:42.529531 25143 server_base.cc:1034] running on GCE node
I20260810 18:15:42.530241 25143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:42.531301 25143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:42.532467 25143 hybrid_clock.cc:648] HybridClock initialized: now 1786385742532439 us; error 42 us; skew 500 ppm
Aug 10 18:15:42 dist-test-slave-jpj1 krb5kdc[25121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:42.539167 25143 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:42.540134 25143 webserver.cc:460] Webserver started at http://127.0.0.1:46597/ using document root <none> and password file <none>
I20260810 18:15:42.540675 25143 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:42.540743 25143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:42.540952 25143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:42.542958 25143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "51ab07a02663425a810328cbc764fb8f"
format_stamp: "Formatted at 2026-08-10 18:15:42 on dist-test-slave-jpj1"
server_key: "3675c4084ce050c839f38f0fbb29f508"
server_key_iv: "ccf0ddf89293fd6047bf0d27e5e7c9d0"
server_key_version: "encryptionkey@0"
I20260810 18:15:42.543505 25143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "51ab07a02663425a810328cbc764fb8f"
format_stamp: "Formatted at 2026-08-10 18:15:42 on dist-test-slave-jpj1"
server_key: "3675c4084ce050c839f38f0fbb29f508"
server_key_iv: "ccf0ddf89293fd6047bf0d27e5e7c9d0"
server_key_version: "encryptionkey@0"
I20260810 18:15:42.547084 25143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:15:42.549484 25173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:42.550680 25143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:15:42.550798 25143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "51ab07a02663425a810328cbc764fb8f"
format_stamp: "Formatted at 2026-08-10 18:15:42 on dist-test-slave-jpj1"
server_key: "3675c4084ce050c839f38f0fbb29f508"
server_key_iv: "ccf0ddf89293fd6047bf0d27e5e7c9d0"
server_key_version: "encryptionkey@0"
I20260810 18:15:42.550894 25143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:42.568269 25143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:42.571470 25143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:42.571674 25143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:42.579363 25143 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41085
I20260810 18:15:42.579653 25271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41085 every 8 connection(s)
I20260810 18:15:42.580420 25143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:42.583164 25273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:42.583668 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 25143
I20260810 18:15:42.583807 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:42.584096 23813 external_mini_cluster.cc:1428] Setting key 1c5fee2266ca7ae213d9a5259103df22
I20260810 18:15:42.589936 25273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f: Bootstrap starting.
I20260810 18:15:42.592525 25273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:42.593415 25273 log.cc:824] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:42.593953 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:42.585530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51457 (local address 127.0.0.1:41085)
0810 18:15:42.585949 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:42.585968 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:42.586012 (+ 44us) server_negotiation.cc:407] Connection header received
0810 18:15:42.586820 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:42.586843 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:42.587273 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:42.587660 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:42.588072 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:42.589524 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:42.590458 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:42.591133 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:42.591314 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:42.592014 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:42.592020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:42.592036 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:42.592526 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:42.592595 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:42.592627 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:42.592728 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:42.592826 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 18:15:42.593028 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260810 18:15:42.595487 25273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f: No bootstrap required, opened a new log
I20260810 18:15:42.596650 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:42.599364 25273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ab07a02663425a810328cbc764fb8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260810 18:15:42.599622 25273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:42.599684 25273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ab07a02663425a810328cbc764fb8f, State: Initialized, Role: FOLLOWER
I20260810 18:15:42.600106 25273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ab07a02663425a810328cbc764fb8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260810 18:15:42.600241 25273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:42.600308 25273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:42.600419 25273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:42.601265 25273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ab07a02663425a810328cbc764fb8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260810 18:15:42.601596 25273 leader_election.cc:302] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51ab07a02663425a810328cbc764fb8f; no voters:
I20260810 18:15:42.601892 25273 leader_election.cc:288] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:42.601984 25294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:42.602214 25294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [term 1 LEADER]: Becoming Leader. State: Replica: 51ab07a02663425a810328cbc764fb8f, State: Running, Role: LEADER
I20260810 18:15:42.602258 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:42.598499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39895 (local address 127.0.0.1:41085)
0810 18:15:42.598607 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:42.598611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:42.598780 (+ 169us) server_negotiation.cc:407] Connection header received
0810 18:15:42.598907 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:42.598909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:42.598944 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:42.599004 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:42.599277 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:42.599889 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:42.600776 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:42.601038 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:42.601126 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:42.601830 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:42.601834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:42.601836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:42.601849 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:42.601878 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:42.601888 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:42.601947 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:42.602047 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 18:15:42.602116 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 18:15:42.602569 25294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ab07a02663425a810328cbc764fb8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260810 18:15:42.603193 25273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:42.603790 25294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ab07a02663425a810328cbc764fb8f. Latest consensus state: current_term: 1 leader_uuid: "51ab07a02663425a810328cbc764fb8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ab07a02663425a810328cbc764fb8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } } }
I20260810 18:15:42.603982 25294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:42.604389 25295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51ab07a02663425a810328cbc764fb8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ab07a02663425a810328cbc764fb8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } } }
I20260810 18:15:42.604523 25295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f [sys.catalog]: This master's current role is: LEADER
W20260810 18:15:42.608014 25326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:15:42.608079 25326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:15:42.608151 25310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:42.608821 25310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:42.613765 25310 catalog_manager.cc:1347] Generated new cluster ID: 1180b96cf1bc4186a7a3d5d433b4b0d3
I20260810 18:15:42.613838 25310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:42.628463 25310 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:42.629695 25310 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:42.636616 25310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51ab07a02663425a810328cbc764fb8f: Generated new TSK 0
I20260810 18:15:42.637383 25310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:42.644389 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 25143
2026-08-10T18:15:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1533 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:42 dist-test-slave-jpj1 krb5kdc[25336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:42 dist-test-slave-jpj1 krb5kdc[25336](info): set up 2 sockets
Aug 10 18:15:42 dist-test-slave-jpj1 krb5kdc[25336](info): commencing operation
krb5kdc: starting...
W20260810 18:15:43.811928 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:43 dist-test-slave-jpj1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:43.931321 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:44.043684 25360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:44.043990 25360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:44.044034 25360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:44.048002 25360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:44.048064 25360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:44.048082 25360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:44.048097 25360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:44.048117 25360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:44.053503 25360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:44.054517 25360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:44.055482 25360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:44.063501 25369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:44.063506 25373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:44.063506 25370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:44.063747 25360 server_base.cc:1034] running on GCE node
I20260810 18:15:44.064429 25360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:44.065466 25360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:44.066630 25360 hybrid_clock.cc:648] HybridClock initialized: now 1786385744066614 us; error 27 us; skew 500 ppm
Aug 10 18:15:44 dist-test-slave-jpj1 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:44.069388 25360 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:44.070299 25360 webserver.cc:460] Webserver started at http://127.0.0.1:45993/ using document root <none> and password file <none>
I20260810 18:15:44.070825 25360 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:44.070894 25360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:44.071080 25360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:44.073149 25360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "567a93d519ff4ea29a4619c6b93a851c"
format_stamp: "Formatted at 2026-08-10 18:15:44 on dist-test-slave-jpj1"
server_key: "89dcd37166687fe1e43416657f9731e5"
server_key_iv: "feac991ab22f495062732f6ce0c36d36"
server_key_version: "encryptionkey@0"
I20260810 18:15:44.073706 25360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "567a93d519ff4ea29a4619c6b93a851c"
format_stamp: "Formatted at 2026-08-10 18:15:44 on dist-test-slave-jpj1"
server_key: "89dcd37166687fe1e43416657f9731e5"
server_key_iv: "feac991ab22f495062732f6ce0c36d36"
server_key_version: "encryptionkey@0"
I20260810 18:15:44.077430 25360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:15:44.079717 25393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:44.080911 25360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:44.081028 25360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "567a93d519ff4ea29a4619c6b93a851c"
format_stamp: "Formatted at 2026-08-10 18:15:44 on dist-test-slave-jpj1"
server_key: "89dcd37166687fe1e43416657f9731e5"
server_key_iv: "feac991ab22f495062732f6ce0c36d36"
server_key_version: "encryptionkey@0"
I20260810 18:15:44.081123 25360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:44.085611 25360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:44.088325 25360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:44.088488 25360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:44.095356 25360 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40295
I20260810 18:15:44.095382 25477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40295 every 8 connection(s)
I20260810 18:15:44.096297 25360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:44.097107 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 25360
I20260810 18:15:44.097256 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:44.097549 23813 external_mini_cluster.cc:1428] Setting key a3f6f95b4c4255cbce1e3c4f55bd1bcf
I20260810 18:15:44.099596 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:44.102582 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:44.098979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36479 (local address 127.0.0.1:40295)
0810 18:15:44.099408 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:44.099423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:44.099459 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:15:44.100160 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:44.100178 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:44.100450 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:44.100764 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:44.101475 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:44.101479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:44.101491 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:44.101502 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:44.101554 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:44.101573 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:44.101705 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:44.101774 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 18:15:44.101923 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":137,"threads_started":1}
I20260810 18:15:44.105345 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:44.105691 25478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c: Bootstrap starting.
I20260810 18:15:44.108083 25478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:44.108877 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:44.107288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57461 (local address 127.0.0.1:40295)
0810 18:15:44.107406 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:44.107410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:44.107601 (+ 191us) server_negotiation.cc:407] Connection header received
0810 18:15:44.107757 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:44.107760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:44.107793 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:44.107856 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:44.108519 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:44.108522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:44.108525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:44.108537 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:44.108563 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:44.108573 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:44.108640 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:44.108722 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 18:15:44.108758 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:15:44.108876 25478 log.cc:824] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:44.110793 25478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c: No bootstrap required, opened a new log
I20260810 18:15:44.114292 25478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567a93d519ff4ea29a4619c6b93a851c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260810 18:15:44.114524 25478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:44.114583 25478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567a93d519ff4ea29a4619c6b93a851c, State: Initialized, Role: FOLLOWER
I20260810 18:15:44.114993 25478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567a93d519ff4ea29a4619c6b93a851c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260810 18:15:44.115120 25478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:44.115187 25478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:44.115288 25478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:44.116283 25478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567a93d519ff4ea29a4619c6b93a851c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260810 18:15:44.116585 25478 leader_election.cc:302] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 567a93d519ff4ea29a4619c6b93a851c; no voters:
I20260810 18:15:44.116849 25478 leader_election.cc:288] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:44.116964 25497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:44.117165 25497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [term 1 LEADER]: Becoming Leader. State: Replica: 567a93d519ff4ea29a4619c6b93a851c, State: Running, Role: LEADER
I20260810 18:15:44.117578 25497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567a93d519ff4ea29a4619c6b93a851c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260810 18:15:44.118018 25478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:44.119190 25499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 567a93d519ff4ea29a4619c6b93a851c. Latest consensus state: current_term: 1 leader_uuid: "567a93d519ff4ea29a4619c6b93a851c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567a93d519ff4ea29a4619c6b93a851c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } } }
I20260810 18:15:44.119334 25499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:44.119740 25509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:44.119699 25498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "567a93d519ff4ea29a4619c6b93a851c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567a93d519ff4ea29a4619c6b93a851c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } } }
I20260810 18:15:44.119917 25498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:44.122813 25509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:44.128273 25509 catalog_manager.cc:1347] Generated new cluster ID: d9c266c4a7e8408b91a34d3a7e5de09d
I20260810 18:15:44.128347 25509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:44.138576 25509 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:44.138861 25509 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:44.152014 25509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 567a93d519ff4ea29a4619c6b93a851c: Generated new TSK 0
I20260810 18:15:44.152801 25509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:44.163512 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 25360
2026-08-10T18:15:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1513 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:44 dist-test-slave-jpj1 krb5kdc[25528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:44 dist-test-slave-jpj1 krb5kdc[25528](info): set up 2 sockets
Aug 10 18:15:44 dist-test-slave-jpj1 krb5kdc[25528](info): commencing operation
krb5kdc: starting...
W20260810 18:15:45.322384 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:45 dist-test-slave-jpj1 krb5kdc[25528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:45.443832 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:45.553017 25550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:45.553273 25550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:45.553316 25550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:45.557230 25550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:45.557287 25550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:45.557305 25550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:45.557324 25550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:45.557353 25550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:45.562552 25550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:45.563625 25550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:45.564631 25550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:45.572835 25556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:45.572825 25557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:45.572824 25560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:45.572949 25550 server_base.cc:1034] running on GCE node
I20260810 18:15:45.573694 25550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:45.574741 25550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:45.575899 25550 hybrid_clock.cc:648] HybridClock initialized: now 1786385745575883 us; error 29 us; skew 500 ppm
Aug 10 18:15:45 dist-test-slave-jpj1 krb5kdc[25528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:45.578495 25550 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:45.579293 25550 webserver.cc:460] Webserver started at http://127.0.0.1:34697/ using document root <none> and password file <none>
I20260810 18:15:45.579834 25550 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:45.579895 25550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:45.580065 25550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:45.581995 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "d358718ce09f4bb88565bf0db1ca3039"
format_stamp: "Formatted at 2026-08-10 18:15:45 on dist-test-slave-jpj1"
server_key: "54e5533f36c6b42dd810adaffe468820"
server_key_iv: "d92abe96c600daf31654a0ccc2cb0fe3"
server_key_version: "encryptionkey@0"
I20260810 18:15:45.582507 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "d358718ce09f4bb88565bf0db1ca3039"
format_stamp: "Formatted at 2026-08-10 18:15:45 on dist-test-slave-jpj1"
server_key: "54e5533f36c6b42dd810adaffe468820"
server_key_iv: "d92abe96c600daf31654a0ccc2cb0fe3"
server_key_version: "encryptionkey@0"
I20260810 18:15:45.586841 25550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:15:45.589258 25574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:45.590451 25550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:15:45.590566 25550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "d358718ce09f4bb88565bf0db1ca3039"
format_stamp: "Formatted at 2026-08-10 18:15:45 on dist-test-slave-jpj1"
server_key: "54e5533f36c6b42dd810adaffe468820"
server_key_iv: "d92abe96c600daf31654a0ccc2cb0fe3"
server_key_version: "encryptionkey@0"
I20260810 18:15:45.590659 25550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:45.608896 25550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:45.611986 25550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:45.612171 25550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:45.618948 25550 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35599
I20260810 18:15:45.618993 25661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35599 every 8 connection(s)
I20260810 18:15:45.619979 25550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:45.622717 25664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:45.628240 25664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039: Bootstrap starting.
I20260810 18:15:45.629892 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 25550
I20260810 18:15:45.630008 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:45.630254 23813 external_mini_cluster.cc:1428] Setting key 7ecf79151cec9e07f23a8785d46ca20a
I20260810 18:15:45.630726 25664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:45.631644 25664 log.cc:824] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:45.633678 25664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039: No bootstrap required, opened a new log
I20260810 18:15:45.637771 25664 raft_consensus.cc:348] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358718ce09f4bb88565bf0db1ca3039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } }
I20260810 18:15:45.638080 25664 raft_consensus.cc:374] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:45.638154 25664 raft_consensus.cc:729] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d358718ce09f4bb88565bf0db1ca3039, State: Initialized, Role: FOLLOWER
I20260810 18:15:45.638633 25664 consensus_queue.cc:260] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358718ce09f4bb88565bf0db1ca3039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } }
I20260810 18:15:45.638767 25664 raft_consensus.cc:388] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:45.638830 25664 raft_consensus.cc:482] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:45.638918 25664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 18:15:45 dist-test-slave-jpj1 krb5kdc[25528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:15:45.639889 25664 raft_consensus.cc:504] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358718ce09f4bb88565bf0db1ca3039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } }
I20260810 18:15:45.640264 25664 leader_election.cc:302] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d358718ce09f4bb88565bf0db1ca3039; no voters:
I20260810 18:15:45.640599 25664 leader_election.cc:288] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:45.640974 25676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:45.641215 25676 raft_consensus.cc:686] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [term 1 LEADER]: Becoming Leader. State: Replica: d358718ce09f4bb88565bf0db1ca3039, State: Running, Role: LEADER
I20260810 18:15:45.641564 25676 consensus_queue.cc:237] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358718ce09f4bb88565bf0db1ca3039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } }
I20260810 18:15:45.641903 25664 sys_catalog.cc:564] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:45.642784 25676 sys_catalog.cc:455] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d358718ce09f4bb88565bf0db1ca3039. Latest consensus state: current_term: 1 leader_uuid: "d358718ce09f4bb88565bf0db1ca3039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358718ce09f4bb88565bf0db1ca3039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } } }
I20260810 18:15:45.642899 25676 sys_catalog.cc:458] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:45.643445 25675 sys_catalog.cc:455] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d358718ce09f4bb88565bf0db1ca3039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358718ce09f4bb88565bf0db1ca3039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } } }
I20260810 18:15:45.643626 25675 sys_catalog.cc:458] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:45.643980 25692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:45.645196 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:45.631330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:47575 (local address 127.0.0.1:35599)
0810 18:15:45.631751 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:45.631770 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:45.631806 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:15:45.632573 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:45.632594 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:45.632940 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:45.633216 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:45.634116 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:45.635072 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:45.636327 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:45.636803 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:45.636927 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:45.639754 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:45.639759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:45.639771 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:45.639799 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:45.641767 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:45.642392 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:45.642398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:45.642404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:45.642541 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:45.642974 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:45.642978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:45.642980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:45.643413 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:45.643665 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:45.644256 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 18:15:45.644483 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":127,"threads_started":1}
I20260810 18:15:45.647222 25692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:45.648659 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:45.653641 25692 catalog_manager.cc:1347] Generated new cluster ID: 551dfc859f184c6f8f19edffe14dfe03
I20260810 18:15:45.653707 25692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:45.666937 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:45.652013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:43361 (local address 127.0.0.1:35599)
0810 18:15:45.652112 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:45.652116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:45.652502 (+ 386us) server_negotiation.cc:407] Connection header received
0810 18:15:45.652669 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:45.652672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:45.652721 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:45.652804 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:45.659680 (+ 6876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:45.660507 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:45.661633 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:45.661993 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:45.662099 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:45.663658 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:45.663663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:45.663665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:45.663691 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:45.665221 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:45.665728 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:45.665732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:45.665734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:45.665819 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:45.666091 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:45.666094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:45.666096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:45.666301 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:45.666447 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:45.666679 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 18:15:45.666776 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 18:15:45.675050 25692 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:45.676304 25692 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:45.684990 25692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d358718ce09f4bb88565bf0db1ca3039: Generated new TSK 0
I20260810 18:15:45.685616 25692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:45.694334 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 25550
W20260810 18:15:45.710271 25524 connection.cc:535] client connection to 127.0.0.1:35599 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35599): Connection reset by peer (error 104)
2026-08-10T18:15:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1533 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:45 dist-test-slave-jpj1 krb5kdc[25720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:45 dist-test-slave-jpj1 krb5kdc[25720](info): set up 2 sockets
Aug 10 18:15:45 dist-test-slave-jpj1 krb5kdc[25720](info): commencing operation
krb5kdc: starting...
W20260810 18:15:46.847483 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:46 dist-test-slave-jpj1 krb5kdc[25720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:46.975335 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:47.087769 25746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:47.088035 25746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:47.088075 25746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:47.091949 25746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:47.092002 25746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:47.092020 25746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:47.092036 25746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:47.092053 25746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:47.097191 25746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:47.098174 25746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:47.099125 25746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:47.107278 25755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:47.107384 25746 server_base.cc:1034] running on GCE node
W20260810 18:15:47.107280 25757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:47.107287 25760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:47.108217 25746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:47.109284 25746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:47.110455 25746 hybrid_clock.cc:648] HybridClock initialized: now 1786385747110440 us; error 26 us; skew 500 ppm
Aug 10 18:15:47 dist-test-slave-jpj1 krb5kdc[25720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:47.113240 25746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:47.114109 25746 webserver.cc:460] Webserver started at http://127.0.0.1:37107/ using document root <none> and password file <none>
I20260810 18:15:47.114622 25746 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:47.114684 25746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:47.114885 25746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:47.116901 25746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "7569f24d7aff4d5a92d29f0f6d117a6a"
format_stamp: "Formatted at 2026-08-10 18:15:47 on dist-test-slave-jpj1"
server_key: "a786b8324279aa70b6710067a44c4cad"
server_key_iv: "a0a6757823c177ce9f330440f8c60276"
server_key_version: "encryptionkey@0"
I20260810 18:15:47.117446 25746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "7569f24d7aff4d5a92d29f0f6d117a6a"
format_stamp: "Formatted at 2026-08-10 18:15:47 on dist-test-slave-jpj1"
server_key: "a786b8324279aa70b6710067a44c4cad"
server_key_iv: "a0a6757823c177ce9f330440f8c60276"
server_key_version: "encryptionkey@0"
I20260810 18:15:47.121017 25746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:15:47.123256 25777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:47.124469 25746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:47.124576 25746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "7569f24d7aff4d5a92d29f0f6d117a6a"
format_stamp: "Formatted at 2026-08-10 18:15:47 on dist-test-slave-jpj1"
server_key: "a786b8324279aa70b6710067a44c4cad"
server_key_iv: "a0a6757823c177ce9f330440f8c60276"
server_key_version: "encryptionkey@0"
I20260810 18:15:47.124673 25746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:47.157882 25746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:47.160962 25746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:47.161142 25746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:47.168071 25746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40903
I20260810 18:15:47.168123 25869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40903 every 8 connection(s)
I20260810 18:15:47.169000 25746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:47.171756 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:47.172668 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 25746
I20260810 18:15:47.172791 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:47.173044 23813 external_mini_cluster.cc:1428] Setting key 8dac92186853805a9c5b2a4d8e666687
I20260810 18:15:47.178709 25870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a: Bootstrap starting.
I20260810 18:15:47.181010 25870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:47.181756 25870 log.cc:824] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:47.181867 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:47.174312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:48495 (local address 127.0.0.1:40903)
0810 18:15:47.174700 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:47.174716 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:47.174752 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:15:47.175615 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:47.175635 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:47.175988 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:47.176303 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:47.176711 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:47.177690 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:47.178578 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:47.179080 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:47.179231 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:47.180148 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:47.180152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:47.180165 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:47.180558 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:47.180612 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:47.180635 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:47.180755 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:47.180941 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 18:15:47.181080 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260810 18:15:47.183878 25870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a: No bootstrap required, opened a new log
I20260810 18:15:47.184769 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:47.187803 25870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569f24d7aff4d5a92d29f0f6d117a6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40903 } }
I20260810 18:15:47.188028 25870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:47.188064 25870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7569f24d7aff4d5a92d29f0f6d117a6a, State: Initialized, Role: FOLLOWER
I20260810 18:15:47.188484 25870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569f24d7aff4d5a92d29f0f6d117a6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40903 } }
I20260810 18:15:47.188611 25870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:47.188673 25870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:47.188763 25870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:47.189679 25870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569f24d7aff4d5a92d29f0f6d117a6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40903 } }
I20260810 18:15:47.190026 25870 leader_election.cc:302] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7569f24d7aff4d5a92d29f0f6d117a6a; no voters:
I20260810 18:15:47.190354 25870 leader_election.cc:288] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:47.190433 25885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:47.190582 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:47.186651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:58871 (local address 127.0.0.1:40903)
0810 18:15:47.186754 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:47.186758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:47.186930 (+ 172us) server_negotiation.cc:407] Connection header received
0810 18:15:47.187041 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:47.187044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:47.187089 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:47.187210 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:47.187456 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:47.188008 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:47.188918 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:47.189172 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:47.189259 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:47.190109 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:47.190113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:47.190116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:47.190129 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:47.190156 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:47.190170 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:47.190247 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:47.190374 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 18:15:47.190406 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 18:15:47.190687 25885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [term 1 LEADER]: Becoming Leader. State: Replica: 7569f24d7aff4d5a92d29f0f6d117a6a, State: Running, Role: LEADER
I20260810 18:15:47.191021 25885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569f24d7aff4d5a92d29f0f6d117a6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40903 } }
I20260810 18:15:47.191859 25870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:47.192258 25885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7569f24d7aff4d5a92d29f0f6d117a6a. Latest consensus state: current_term: 1 leader_uuid: "7569f24d7aff4d5a92d29f0f6d117a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569f24d7aff4d5a92d29f0f6d117a6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40903 } } }
I20260810 18:15:47.192370 25885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:47.192603 25885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7569f24d7aff4d5a92d29f0f6d117a6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569f24d7aff4d5a92d29f0f6d117a6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40903 } } }
I20260810 18:15:47.192763 25893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:47.192778 25885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:47.195681 25893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:47.201081 25893 catalog_manager.cc:1347] Generated new cluster ID: c9001213bbe34b8c9e3eda33bc074f56
I20260810 18:15:47.201149 25893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:47.217097 25893 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:47.218282 25893 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:47.246767 25893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7569f24d7aff4d5a92d29f0f6d117a6a: Generated new TSK 0
I20260810 18:15:47.247563 25893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:47.257591 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 25746
W20260810 18:15:47.271402 25718 connection.cc:535] client connection to 127.0.0.1:40903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40903): Connection reset by peer (error 104)
2026-08-10T18:15:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1562 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:47 dist-test-slave-jpj1 krb5kdc[25922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:47 dist-test-slave-jpj1 krb5kdc[25922](info): set up 2 sockets
Aug 10 18:15:47 dist-test-slave-jpj1 krb5kdc[25922](info): commencing operation
krb5kdc: starting...
W20260810 18:15:48.422015 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:48 dist-test-slave-jpj1 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:48.540112 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:48.650081 25945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:48.650357 25945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:48.650413 25945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:48.654316 25945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:48.654373 25945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:48.654400 25945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:48.654420 25945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:48.654441 25945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:48.659509 25945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:48.660516 25945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:48.661500 25945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:48.669528 25954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:48.669605 25952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:48.669586 25959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:48.669700 25945 server_base.cc:1034] running on GCE node
I20260810 18:15:48.670481 25945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:48.671571 25945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:48.672736 25945 hybrid_clock.cc:648] HybridClock initialized: now 1786385748672722 us; error 30 us; skew 500 ppm
Aug 10 18:15:48 dist-test-slave-jpj1 krb5kdc[25922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:48.679258 25945 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:48.680147 25945 webserver.cc:460] Webserver started at http://127.0.0.1:35551/ using document root <none> and password file <none>
I20260810 18:15:48.680658 25945 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:48.680718 25945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:48.680900 25945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:48.682889 25945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "5fec440e705c47efbd0971a64ae2c1bd"
format_stamp: "Formatted at 2026-08-10 18:15:48 on dist-test-slave-jpj1"
server_key: "376026d2bb2f45673e915243f6fd1227"
server_key_iv: "79b3cbb5a87a8494dfd90f4e3008cadf"
server_key_version: "encryptionkey@0"
I20260810 18:15:48.683418 25945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "5fec440e705c47efbd0971a64ae2c1bd"
format_stamp: "Formatted at 2026-08-10 18:15:48 on dist-test-slave-jpj1"
server_key: "376026d2bb2f45673e915243f6fd1227"
server_key_iv: "79b3cbb5a87a8494dfd90f4e3008cadf"
server_key_version: "encryptionkey@0"
I20260810 18:15:48.686923 25945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:15:48.689271 25970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:48.690469 25945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:48.690585 25945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "5fec440e705c47efbd0971a64ae2c1bd"
format_stamp: "Formatted at 2026-08-10 18:15:48 on dist-test-slave-jpj1"
server_key: "376026d2bb2f45673e915243f6fd1227"
server_key_iv: "79b3cbb5a87a8494dfd90f4e3008cadf"
server_key_version: "encryptionkey@0"
I20260810 18:15:48.690690 25945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:48.695151 25945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:48.698067 25945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:48.698235 25945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:48.704902 25945 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46843
I20260810 18:15:48.704913 26042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46843 every 8 connection(s)
I20260810 18:15:48.705783 25945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:48.708465 26043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:48.713757 26043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd: Bootstrap starting.
I20260810 18:15:48.715427 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 25945
I20260810 18:15:48.715606 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:48.715910 23813 external_mini_cluster.cc:1428] Setting key 1d4a0cf891056f4d14bb7869dcd7380d
I20260810 18:15:48.716449 26043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:48.717312 26043 log.cc:824] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:48.719280 26043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd: No bootstrap required, opened a new log
I20260810 18:15:48.720685 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:48.716954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:50631 (local address 127.0.0.1:46843)
0810 18:15:48.717349 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:48.717363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:48.717401 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:15:48.718177 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:48.718197 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:48.718463 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:48.718733 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:48.719507 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:48.719511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:48.719522 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:48.719549 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:48.719612 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:48.719634 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:48.719720 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:48.719837 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 18:15:48.719976 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260810 18:15:48.723172 26043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec440e705c47efbd0971a64ae2c1bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260810 18:15:48.723444 26043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:48.723506 26043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fec440e705c47efbd0971a64ae2c1bd, State: Initialized, Role: FOLLOWER
I20260810 18:15:48.723505 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:48.723932 26043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec440e705c47efbd0971a64ae2c1bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260810 18:15:48.724054 26043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:48.724114 26043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:48.724208 26043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:48.725121 26043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec440e705c47efbd0971a64ae2c1bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260810 18:15:48.725453 26043 leader_election.cc:302] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fec440e705c47efbd0971a64ae2c1bd; no voters:
I20260810 18:15:48.725793 26043 leader_election.cc:288] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:48.725878 26057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:48.726114 26057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [term 1 LEADER]: Becoming Leader. State: Replica: 5fec440e705c47efbd0971a64ae2c1bd, State: Running, Role: LEADER
I20260810 18:15:48.726491 26057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec440e705c47efbd0971a64ae2c1bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } }
I20260810 18:15:48.727147 26043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:48.727272 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:48.725592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:32909 (local address 127.0.0.1:46843)
0810 18:15:48.725702 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:48.725706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:48.725881 (+ 175us) server_negotiation.cc:407] Connection header received
0810 18:15:48.726021 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:48.726024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:48.726059 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:48.726131 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:48.726889 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:48.726893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:48.726895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:48.726907 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:48.726936 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:48.726949 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:48.727034 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:48.727120 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 18:15:48.727160 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:15:48.728323 26059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fec440e705c47efbd0971a64ae2c1bd. Latest consensus state: current_term: 1 leader_uuid: "5fec440e705c47efbd0971a64ae2c1bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec440e705c47efbd0971a64ae2c1bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } } }
I20260810 18:15:48.728438 26059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:48.728543 26062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fec440e705c47efbd0971a64ae2c1bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec440e705c47efbd0971a64ae2c1bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46843 } } }
I20260810 18:15:48.728641 26062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:48.728855 26074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:48.732192 26074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:48.737839 26074 catalog_manager.cc:1347] Generated new cluster ID: 369a52b7c38a44e9a78eb9c1bd942a3a
I20260810 18:15:48.737903 26074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:48.750093 26074 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:48.750499 26074 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:48.761134 26074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fec440e705c47efbd0971a64ae2c1bd: Generated new TSK 0
I20260810 18:15:48.761864 26074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:48.770994 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 25945
2026-08-10T18:15:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1515 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:48 dist-test-slave-jpj1 krb5kdc[26094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:48 dist-test-slave-jpj1 krb5kdc[26094](info): set up 2 sockets
Aug 10 18:15:48 dist-test-slave-jpj1 krb5kdc[26094](info): commencing operation
krb5kdc: starting...
W20260810 18:15:49.930265 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:50 dist-test-slave-jpj1 krb5kdc[26094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:50.054517 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:50.166875 26128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:50.167135 26128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:50.167192 26128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:50.171125 26128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:50.171192 26128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:50.171219 26128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:50.171238 26128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:50.171259 26128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:50.176594 26128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:50.177685 26128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:50.178741 26128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:50.187080 26142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:50.187086 26138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:50.187166 26139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:50.187285 26128 server_base.cc:1034] running on GCE node
I20260810 18:15:50.188047 26128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:50.189184 26128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:50.190362 26128 hybrid_clock.cc:648] HybridClock initialized: now 1786385750190338 us; error 33 us; skew 500 ppm
Aug 10 18:15:50 dist-test-slave-jpj1 krb5kdc[26094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:50.193017 26128 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:50.193894 26128 webserver.cc:460] Webserver started at http://127.0.0.1:35397/ using document root <none> and password file <none>
I20260810 18:15:50.194434 26128 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:50.194499 26128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:50.194691 26128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:50.196770 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "e19309ccc2c14da8914d8eef2c8b950e"
format_stamp: "Formatted at 2026-08-10 18:15:50 on dist-test-slave-jpj1"
server_key: "6ec1245b565aadb993049a105d85b02c"
server_key_iv: "aa72cbd4d17438bc7f15accd8d42e72a"
server_key_version: "encryptionkey@0"
I20260810 18:15:50.197319 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "e19309ccc2c14da8914d8eef2c8b950e"
format_stamp: "Formatted at 2026-08-10 18:15:50 on dist-test-slave-jpj1"
server_key: "6ec1245b565aadb993049a105d85b02c"
server_key_iv: "aa72cbd4d17438bc7f15accd8d42e72a"
server_key_version: "encryptionkey@0"
I20260810 18:15:50.201025 26128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:50.203361 26160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:50.204563 26128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:50.204679 26128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "e19309ccc2c14da8914d8eef2c8b950e"
format_stamp: "Formatted at 2026-08-10 18:15:50 on dist-test-slave-jpj1"
server_key: "6ec1245b565aadb993049a105d85b02c"
server_key_iv: "aa72cbd4d17438bc7f15accd8d42e72a"
server_key_version: "encryptionkey@0"
I20260810 18:15:50.204773 26128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:50.220608 26128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:50.223354 26128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:50.223577 26128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:50.230561 26128 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38925
I20260810 18:15:50.230574 26240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38925 every 8 connection(s)
I20260810 18:15:50.231554 26128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:50.234288 26241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:50.239861 26241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e: Bootstrap starting.
I20260810 18:15:50.241257 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 26128
I20260810 18:15:50.241402 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:50.241686 23813 external_mini_cluster.cc:1428] Setting key 44eb0e717c708793b92eb03a77af9a06
I20260810 18:15:50.242663 26241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:50.243469 26241 log.cc:824] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:50.245635 26241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e: No bootstrap required, opened a new log
I20260810 18:15:50.249739 26241 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19309ccc2c14da8914d8eef2c8b950e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260810 18:15:50.250041 26241 raft_consensus.cc:374] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:50.250105 26241 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19309ccc2c14da8914d8eef2c8b950e, State: Initialized, Role: FOLLOWER
I20260810 18:15:50.250627 26241 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19309ccc2c14da8914d8eef2c8b950e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
Aug 10 18:15:50 dist-test-slave-jpj1 krb5kdc[26094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:15:50.250797 26241 raft_consensus.cc:388] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:50.250886 26241 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:50.250991 26241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:50.252053 26241 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19309ccc2c14da8914d8eef2c8b950e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260810 18:15:50.252426 26241 leader_election.cc:302] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e19309ccc2c14da8914d8eef2c8b950e; no voters:
I20260810 18:15:50.252774 26241 leader_election.cc:288] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:50.252882 26251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:50.253108 26251 raft_consensus.cc:686] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [term 1 LEADER]: Becoming Leader. State: Replica: e19309ccc2c14da8914d8eef2c8b950e, State: Running, Role: LEADER
I20260810 18:15:50.253500 26251 consensus_queue.cc:237] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19309ccc2c14da8914d8eef2c8b950e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260810 18:15:50.254065 26241 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:50.255229 26255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e19309ccc2c14da8914d8eef2c8b950e. Latest consensus state: current_term: 1 leader_uuid: "e19309ccc2c14da8914d8eef2c8b950e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19309ccc2c14da8914d8eef2c8b950e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } } }
I20260810 18:15:50.255355 26255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:50.255694 26254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e19309ccc2c14da8914d8eef2c8b950e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19309ccc2c14da8914d8eef2c8b950e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } } }
I20260810 18:15:50.255788 26254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:50.256165 26265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:50.257726 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:50.242825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:60867 (local address 127.0.0.1:38925)
0810 18:15:50.243239 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:50.243258 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:50.243293 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:15:50.244052 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:50.244075 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:50.244464 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:50.244748 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:50.245616 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:50.246623 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:50.247829 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:50.248297 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:50.248411 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:50.251478 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:50.251483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:50.251497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:50.251526 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:50.253602 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:50.254209 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:50.254214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:50.254220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:50.254369 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:50.254768 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:50.254772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:50.254774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:50.255355 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:50.255619 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:50.256544 (+ 925us) server_negotiation.cc:299] Negotiation successful
0810 18:15:50.256809 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":125,"threads_started":1}
I20260810 18:15:50.259377 26265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:50.261238 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:50.265599 26265 catalog_manager.cc:1347] Generated new cluster ID: b89dc47d93264e4ebaed5bbc8088ec13
I20260810 18:15:50.265669 26265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:50.271505 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:50.263591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:37513 (local address 127.0.0.1:38925)
0810 18:15:50.263681 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:50.263686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:50.263956 (+ 270us) server_negotiation.cc:407] Connection header received
0810 18:15:50.264084 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:50.264087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:50.264146 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:50.264241 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:50.265099 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:50.265845 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:50.266637 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:50.266888 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:50.266978 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:50.268461 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:50.268464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:50.268467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:50.268491 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:50.269795 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:50.270278 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:50.270281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:50.270283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:50.270360 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:50.270675 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:50.270678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:50.270680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:50.270850 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:50.270984 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:50.271258 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 18:15:50.271351 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 18:15:50.287480 26265 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:50.288746 26265 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:50.295974 26265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e19309ccc2c14da8914d8eef2c8b950e: Generated new TSK 0
I20260810 18:15:50.296797 26265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:50.308131 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 26128
W20260810 18:15:50.320979 26092 connection.cc:535] client connection to 127.0.0.1:38925 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38925): Connection reset by peer (error 104)
2026-08-10T18:15:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:50 dist-test-slave-jpj1 krb5kdc[26297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:50 dist-test-slave-jpj1 krb5kdc[26297](info): set up 2 sockets
Aug 10 18:15:50 dist-test-slave-jpj1 krb5kdc[26297](info): commencing operation
krb5kdc: starting...
W20260810 18:15:51.476866 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:51 dist-test-slave-jpj1 krb5kdc[26297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:51.598870 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:51.711443 26314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:51.711795 26314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:51.711846 26314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:51.716169 26314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:51.716235 26314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:51.716254 26314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:51.716271 26314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:51.716300 26314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:51.721606 26314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:51.722602 26314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:51.723609 26314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:51.731873 26319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:51.731978 26314 server_base.cc:1034] running on GCE node
W20260810 18:15:51.731864 26320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:51.731873 26325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:51.732748 26314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:51.733806 26314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:51.734961 26314 hybrid_clock.cc:648] HybridClock initialized: now 1786385751734949 us; error 27 us; skew 500 ppm
Aug 10 18:15:51 dist-test-slave-jpj1 krb5kdc[26297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:51.737646 26314 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:51.738509 26314 webserver.cc:460] Webserver started at http://127.0.0.1:44467/ using document root <none> and password file <none>
I20260810 18:15:51.739029 26314 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:51.739090 26314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:51.739286 26314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:51.741369 26314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "f6af807665d548eaa0cf95ee151ec5b7"
format_stamp: "Formatted at 2026-08-10 18:15:51 on dist-test-slave-jpj1"
server_key: "cb3fc47eb3717c2edbe2a83d72f63031"
server_key_iv: "c560a3fb67f2ab11a97df65a81906650"
server_key_version: "encryptionkey@0"
I20260810 18:15:51.741904 26314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "f6af807665d548eaa0cf95ee151ec5b7"
format_stamp: "Formatted at 2026-08-10 18:15:51 on dist-test-slave-jpj1"
server_key: "cb3fc47eb3717c2edbe2a83d72f63031"
server_key_iv: "c560a3fb67f2ab11a97df65a81906650"
server_key_version: "encryptionkey@0"
I20260810 18:15:51.745550 26314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:15:51.747821 26342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:51.748976 26314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:15:51.749099 26314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "f6af807665d548eaa0cf95ee151ec5b7"
format_stamp: "Formatted at 2026-08-10 18:15:51 on dist-test-slave-jpj1"
server_key: "cb3fc47eb3717c2edbe2a83d72f63031"
server_key_iv: "c560a3fb67f2ab11a97df65a81906650"
server_key_version: "encryptionkey@0"
I20260810 18:15:51.749190 26314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:51.762336 26314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:51.764951 26314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:51.765118 26314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:51.771796 26314 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45353
I20260810 18:15:51.771839 26418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45353 every 8 connection(s)
I20260810 18:15:51.772732 26314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:51.775056 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 26314
I20260810 18:15:51.775202 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:51.775523 23813 external_mini_cluster.cc:1428] Setting key e115ee54995b5604f1c8821758dc1a1b
I20260810 18:15:51.775524 26419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:51.782599 26419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7: Bootstrap starting.
I20260810 18:15:51.785138 26419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:51.785336 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:51.776924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:55633 (local address 127.0.0.1:45353)
0810 18:15:51.777545 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:51.777566 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:51.777610 (+ 44us) server_negotiation.cc:407] Connection header received
0810 18:15:51.778249 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:51.778271 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:51.778699 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:51.779070 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:51.779466 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:51.780952 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:51.781809 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:51.782469 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:51.782620 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:51.783402 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:51.783408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:51.783422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:51.783943 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:51.784009 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:51.784038 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:51.784164 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:51.784297 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 18:15:51.784421 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":139,"threads_started":1}
I20260810 18:15:51.785941 26419 log.cc:824] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:51.787984 26419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7: No bootstrap required, opened a new log
I20260810 18:15:51.788151 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:51.791625 26419 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6af807665d548eaa0cf95ee151ec5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45353 } }
I20260810 18:15:51.791857 26419 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:51.791916 26419 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6af807665d548eaa0cf95ee151ec5b7, State: Initialized, Role: FOLLOWER
I20260810 18:15:51.792322 26419 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6af807665d548eaa0cf95ee151ec5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45353 } }
I20260810 18:15:51.792464 26419 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:51.792531 26419 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:51.792629 26419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:51.793452 26419 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6af807665d548eaa0cf95ee151ec5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45353 } }
I20260810 18:15:51.793593 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:51.789967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:40339 (local address 127.0.0.1:45353)
0810 18:15:51.790061 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:51.790066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:51.790263 (+ 197us) server_negotiation.cc:407] Connection header received
0810 18:15:51.790376 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:51.790379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:51.790419 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:51.790481 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:51.790726 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:51.791200 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:51.792094 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:51.792312 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:51.792401 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:51.793158 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:51.793162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:51.793163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:51.793174 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:51.793197 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:51.793207 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:51.793272 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:51.793431 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 18:15:51.793463 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 18:15:51.793807 26419 leader_election.cc:302] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6af807665d548eaa0cf95ee151ec5b7; no voters:
I20260810 18:15:51.794109 26419 leader_election.cc:288] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:51.794171 26436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:51.794370 26436 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [term 1 LEADER]: Becoming Leader. State: Replica: f6af807665d548eaa0cf95ee151ec5b7, State: Running, Role: LEADER
I20260810 18:15:51.794708 26436 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6af807665d548eaa0cf95ee151ec5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45353 } }
I20260810 18:15:51.795342 26419 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:51.795897 26439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6af807665d548eaa0cf95ee151ec5b7. Latest consensus state: current_term: 1 leader_uuid: "f6af807665d548eaa0cf95ee151ec5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6af807665d548eaa0cf95ee151ec5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45353 } } }
I20260810 18:15:51.796025 26439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:51.796128 26437 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6af807665d548eaa0cf95ee151ec5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6af807665d548eaa0cf95ee151ec5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45353 } } }
I20260810 18:15:51.796494 26437 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:51.796542 26451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:51.799181 26451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:51.804759 26451 catalog_manager.cc:1347] Generated new cluster ID: 0a4d96ab11614cc4873c4cf55065c609
I20260810 18:15:51.805079 26451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:51.831187 26451 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:51.832533 26451 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:51.837698 26451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6af807665d548eaa0cf95ee151ec5b7: Generated new TSK 0
I20260810 18:15:51.838370 26451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:51.852042 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 26314
W20260810 18:15:51.869745 26294 connection.cc:535] client connection to 127.0.0.1:45353 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45353): Connection reset by peer (error 104)
2026-08-10T18:15:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1548 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:51 dist-test-slave-jpj1 krb5kdc[26466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:51 dist-test-slave-jpj1 krb5kdc[26466](info): set up 2 sockets
Aug 10 18:15:51 dist-test-slave-jpj1 krb5kdc[26466](info): commencing operation
krb5kdc: starting...
W20260810 18:15:53.036293 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:53 dist-test-slave-jpj1 krb5kdc[26466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:15:53.162400 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:53.275933 26487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:53.276196 26487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:53.276237 26487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:53.280105 26487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:53.280160 26487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:53.280179 26487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:53.280193 26487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:53.280207 26487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:53.285645 26487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:53.286603 26487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:53.287585 26487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:53.295709 26498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:53.295720 26494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:53.295738 26496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:53.296030 26487 server_base.cc:1034] running on GCE node
I20260810 18:15:53.296581 26487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:53.297667 26487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:53.298835 26487 hybrid_clock.cc:648] HybridClock initialized: now 1786385753298809 us; error 38 us; skew 500 ppm
Aug 10 18:15:53 dist-test-slave-jpj1 krb5kdc[26466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:53.301489 26487 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:15:53.302387 26487 webserver.cc:460] Webserver started at http://127.0.0.1:35985/ using document root <none> and password file <none>
I20260810 18:15:53.302907 26487 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:53.302965 26487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:53.303125 26487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:53.305142 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "9680eb07b73c417d979c8ba74339ef89"
format_stamp: "Formatted at 2026-08-10 18:15:53 on dist-test-slave-jpj1"
server_key: "e4b451992d81efc8a812ab9e1b21756b"
server_key_iv: "9f93779c58ef7da9fb56d192b063a02b"
server_key_version: "encryptionkey@0"
I20260810 18:15:53.305649 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "9680eb07b73c417d979c8ba74339ef89"
format_stamp: "Formatted at 2026-08-10 18:15:53 on dist-test-slave-jpj1"
server_key: "e4b451992d81efc8a812ab9e1b21756b"
server_key_iv: "9f93779c58ef7da9fb56d192b063a02b"
server_key_version: "encryptionkey@0"
I20260810 18:15:53.309228 26487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 18:15:53.311523 26507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:53.312721 26487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:15:53.312822 26487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "9680eb07b73c417d979c8ba74339ef89"
format_stamp: "Formatted at 2026-08-10 18:15:53 on dist-test-slave-jpj1"
server_key: "e4b451992d81efc8a812ab9e1b21756b"
server_key_iv: "9f93779c58ef7da9fb56d192b063a02b"
server_key_version: "encryptionkey@0"
I20260810 18:15:53.312904 26487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:53.317247 26487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:53.320106 26487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:53.320250 26487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:53.326967 26487 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38249
I20260810 18:15:53.327036 26601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38249 every 8 connection(s)
I20260810 18:15:53.327912 26487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:53.329116 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 26487
I20260810 18:15:53.329255 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:53.329571 23813 external_mini_cluster.cc:1428] Setting key ce9e7bb307abc5e2823881b4310b5f41
I20260810 18:15:53.331077 26603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:53.334549 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:53.330966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:42623 (local address 127.0.0.1:38249)
0810 18:15:53.331385 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:53.331399 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:53.331436 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:15:53.332121 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:53.332138 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:53.332416 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:53.332723 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:53.333436 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:53.333441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:53.333452 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:53.333462 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:53.333514 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:53.333533 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:53.333656 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:53.333723 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 18:15:53.333887 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":140,"threads_started":1}
I20260810 18:15:53.337040 26603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89: Bootstrap starting.
I20260810 18:15:53.337208 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:53.339272 26603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:53.340188 26603 log.cc:824] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:53.340790 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:53.339143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:60471 (local address 127.0.0.1:38249)
0810 18:15:53.339263 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:53.339268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:53.339461 (+ 193us) server_negotiation.cc:407] Connection header received
0810 18:15:53.339618 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:53.339621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:53.339654 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:53.339714 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:53.340397 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:53.340400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:53.340401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:53.340414 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:53.340438 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:53.340448 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:53.340514 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:53.340637 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 18:15:53.340677 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:15:53.342303 26603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89: No bootstrap required, opened a new log
I20260810 18:15:53.345703 26603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9680eb07b73c417d979c8ba74339ef89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260810 18:15:53.345913 26603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:53.345947 26603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9680eb07b73c417d979c8ba74339ef89, State: Initialized, Role: FOLLOWER
I20260810 18:15:53.346379 26603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9680eb07b73c417d979c8ba74339ef89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260810 18:15:53.346546 26603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:53.346614 26603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:53.346709 26603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:53.347591 26603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9680eb07b73c417d979c8ba74339ef89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260810 18:15:53.347898 26603 leader_election.cc:302] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9680eb07b73c417d979c8ba74339ef89; no voters:
I20260810 18:15:53.348169 26603 leader_election.cc:288] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:53.348306 26618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:53.348511 26618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [term 1 LEADER]: Becoming Leader. State: Replica: 9680eb07b73c417d979c8ba74339ef89, State: Running, Role: LEADER
I20260810 18:15:53.348834 26618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9680eb07b73c417d979c8ba74339ef89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260810 18:15:53.349323 26603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:53.350360 26621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9680eb07b73c417d979c8ba74339ef89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9680eb07b73c417d979c8ba74339ef89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } } }
I20260810 18:15:53.350494 26621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:53.350790 26623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9680eb07b73c417d979c8ba74339ef89. Latest consensus state: current_term: 1 leader_uuid: "9680eb07b73c417d979c8ba74339ef89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9680eb07b73c417d979c8ba74339ef89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } } }
I20260810 18:15:53.350905 26623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:53.350889 26632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:53.354354 26632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:53.359828 26632 catalog_manager.cc:1347] Generated new cluster ID: 590f1b4c62494a1396610cdacc718b96
I20260810 18:15:53.359884 26632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:53.370111 26632 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:53.370404 26632 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:53.378538 26632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9680eb07b73c417d979c8ba74339ef89: Generated new TSK 0
I20260810 18:15:53.379211 26632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:53.390368 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 26487
2026-08-10T18:15:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:53 dist-test-slave-jpj1 krb5kdc[26657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:53 dist-test-slave-jpj1 krb5kdc[26657](info): set up 2 sockets
Aug 10 18:15:53 dist-test-slave-jpj1 krb5kdc[26657](info): commencing operation
krb5kdc: starting...
W20260810 18:15:54.544950 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:54 dist-test-slave-jpj1 krb5kdc[26657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:15:54.668071 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:44743
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:37487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:44743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:54.778460 26676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:54.778716 26676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:54.778757 26676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:54.783020 26676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:54.783077 26676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:54.783097 26676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:54.783116 26676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:54.783143 26676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:54.788354 26676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:37487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:44743
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:44743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:54.789318 26676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:54.790302 26676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:15:54.798527 26676 server_base.cc:1034] running on GCE node
W20260810 18:15:54.798497 26690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:54.798477 26683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:54.798473 26684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:54.799322 26676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:54.800400 26676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:54.801564 26676 hybrid_clock.cc:648] HybridClock initialized: now 1786385754801550 us; error 24 us; skew 500 ppm
Aug 10 18:15:54 dist-test-slave-jpj1 krb5kdc[26657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385754, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:54.808543 26676 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:15:54.809414 26676 webserver.cc:460] Webserver started at http://127.23.65.126:33907/ using document root <none> and password file <none>
I20260810 18:15:54.809943 26676 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:54.810003 26676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:54.810186 26676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:54.812265 26676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "77a339f20a39421c839b3b0d3c291fe9"
format_stamp: "Formatted at 2026-08-10 18:15:54 on dist-test-slave-jpj1"
server_key: "57233811e4ac5e2f51fe1cda24872258"
server_key_iv: "a6ca7f840deb1b02d11f7d4b3537ac3e"
server_key_version: "encryptionkey@0"
I20260810 18:15:54.812784 26676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "77a339f20a39421c839b3b0d3c291fe9"
format_stamp: "Formatted at 2026-08-10 18:15:54 on dist-test-slave-jpj1"
server_key: "57233811e4ac5e2f51fe1cda24872258"
server_key_iv: "a6ca7f840deb1b02d11f7d4b3537ac3e"
server_key_version: "encryptionkey@0"
I20260810 18:15:54.816448 26676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:54.818727 26702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:54.819932 26676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:54.820047 26676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "77a339f20a39421c839b3b0d3c291fe9"
format_stamp: "Formatted at 2026-08-10 18:15:54 on dist-test-slave-jpj1"
server_key: "57233811e4ac5e2f51fe1cda24872258"
server_key_iv: "a6ca7f840deb1b02d11f7d4b3537ac3e"
server_key_version: "encryptionkey@0"
I20260810 18:15:54.820139 26676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:54.837471 26676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:54.840533 26676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:54.840725 26676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:54.847695 26676 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:44743
I20260810 18:15:54.847715 26785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:44743 every 8 connection(s)
I20260810 18:15:54.848590 26676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:54.851344 26786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:54.854570 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 26676
I20260810 18:15:54.854722 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:54.855041 23813 external_mini_cluster.cc:1428] Setting key 7d09123bce8674057bd436f00ead0872
I20260810 18:15:54.857316 26786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9: Bootstrap starting.
I20260810 18:15:54.860011 26786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:54.860888 26786 log.cc:824] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:54.862998 26786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9: No bootstrap required, opened a new log
Aug 10 18:15:54 dist-test-slave-jpj1 krb5kdc[26657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:54.867692 26786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a339f20a39421c839b3b0d3c291fe9" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 44743 } }
I20260810 18:15:54.867976 26786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:54.868081 26786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a339f20a39421c839b3b0d3c291fe9, State: Initialized, Role: FOLLOWER
I20260810 18:15:54.868582 26786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a339f20a39421c839b3b0d3c291fe9" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 44743 } }
I20260810 18:15:54.868798 26786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:54.868901 26786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:54.869016 26786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:54.869990 26786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a339f20a39421c839b3b0d3c291fe9" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 44743 } }
I20260810 18:15:54.870342 26786 leader_election.cc:302] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77a339f20a39421c839b3b0d3c291fe9; no voters:
I20260810 18:15:54.870718 26786 leader_election.cc:288] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:54.870771 26793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:54.871013 26793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 77a339f20a39421c839b3b0d3c291fe9, State: Running, Role: LEADER
I20260810 18:15:54.871371 26793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a339f20a39421c839b3b0d3c291fe9" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 44743 } }
I20260810 18:15:54.871718 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:54.856409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36273 (local address 127.23.65.126:44743)
0810 18:15:54.856799 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:54.856816 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:54.856853 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:15:54.857479 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:54.857497 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:54.857891 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:54.858257 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:54.859598 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:54.861154 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:54.862367 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:54.863070 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:54.863243 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:54.866377 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:54.866384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:54.866402 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:54.866434 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:54.868396 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:54.868989 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:54.868995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:54.869001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:54.869178 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:54.869568 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:54.869572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:54.869573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:54.869980 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:54.870193 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:54.870506 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 18:15:54.870766 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
I20260810 18:15:54.872046 26786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:54.872663 26793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77a339f20a39421c839b3b0d3c291fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a339f20a39421c839b3b0d3c291fe9" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 44743 } } }
I20260810 18:15:54.872848 26793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:54.873065 26794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77a339f20a39421c839b3b0d3c291fe9. Latest consensus state: current_term: 1 leader_uuid: "77a339f20a39421c839b3b0d3c291fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a339f20a39421c839b3b0d3c291fe9" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 44743 } } }
I20260810 18:15:54.873175 26794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:54.873251 26806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:54.875197 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:54.876214 26806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:54.882352 26806 catalog_manager.cc:1347] Generated new cluster ID: f154a871876648029b6c2e55569f3876
I20260810 18:15:54.882468 26806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:54.887593 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:54.877369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49687 (local address 127.23.65.126:44743)
0810 18:15:54.877564 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:54.877568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:54.878396 (+ 828us) server_negotiation.cc:407] Connection header received
0810 18:15:54.878501 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:54.878505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:54.878588 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:54.878704 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:54.880257 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:54.880857 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:54.882040 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:54.882405 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:54.882485 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:54.884303 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:54.884307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:54.884310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:54.884335 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:54.885888 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:54.886392 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:54.886395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:54.886397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:54.886470 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:54.886775 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:54.886778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:54.886779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:54.886953 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:54.887077 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:54.887336 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 18:15:54.887424 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260810 18:15:54.893325 26806 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:54.894456 26806 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:54.900847 26806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77a339f20a39421c839b3b0d3c291fe9: Generated new TSK 0
I20260810 18:15:54.901595 26806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:54.905791 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 26676
W20260810 18:15:54.927357 26654 connection.cc:535] client connection to 127.23.65.126:44743 recv error: Network error: failed to read from TLS socket (remote: 127.23.65.126:44743): Connection reset by peer (error 104)
2026-08-10T18:15:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:54 dist-test-slave-jpj1 krb5kdc[26840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:54 dist-test-slave-jpj1 krb5kdc[26840](info): set up 2 sockets
Aug 10 18:15:54 dist-test-slave-jpj1 krb5kdc[26840](info): commencing operation
krb5kdc: starting...
W20260810 18:15:56.065086 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:56 dist-test-slave-jpj1 krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:15:56.188431 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:36309
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:36309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:56.301594 26866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:56.301874 26866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:56.301934 26866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:56.305872 26866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:56.305927 26866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:56.305953 26866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:56.305974 26866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:56.306000 26866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:56.311121 26866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:36309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:36309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:56.312156 26866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:56.313138 26866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:56.321132 26878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:56.321125 26879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:56.321123 26882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:56.321444 26866 server_base.cc:1034] running on GCE node
I20260810 18:15:56.322012 26866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:56.323020 26866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:56.324169 26866 hybrid_clock.cc:648] HybridClock initialized: now 1786385756324153 us; error 27 us; skew 500 ppm
Aug 10 18:15:56 dist-test-slave-jpj1 krb5kdc[26840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385756, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:56.326733 26866 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:15:56.327615 26866 webserver.cc:460] Webserver started at http://127.23.65.126:41593/ using document root <none> and password file <none>
I20260810 18:15:56.328138 26866 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:56.328203 26866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:56.328387 26866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:56.330391 26866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "af1b44900a6f4bd4a0bff7ed92b4421a"
format_stamp: "Formatted at 2026-08-10 18:15:56 on dist-test-slave-jpj1"
server_key: "fae017dbf1ec1ea7c88f1e8f9296d71b"
server_key_iv: "3fc0e18c0ee078ab8fcb27517018f40e"
server_key_version: "encryptionkey@0"
I20260810 18:15:56.330932 26866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "af1b44900a6f4bd4a0bff7ed92b4421a"
format_stamp: "Formatted at 2026-08-10 18:15:56 on dist-test-slave-jpj1"
server_key: "fae017dbf1ec1ea7c88f1e8f9296d71b"
server_key_iv: "3fc0e18c0ee078ab8fcb27517018f40e"
server_key_version: "encryptionkey@0"
I20260810 18:15:56.334458 26866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:15:56.336781 26895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:56.338042 26866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:15:56.338167 26866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "af1b44900a6f4bd4a0bff7ed92b4421a"
format_stamp: "Formatted at 2026-08-10 18:15:56 on dist-test-slave-jpj1"
server_key: "fae017dbf1ec1ea7c88f1e8f9296d71b"
server_key_iv: "3fc0e18c0ee078ab8fcb27517018f40e"
server_key_version: "encryptionkey@0"
I20260810 18:15:56.338266 26866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:56.355408 26866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:56.358114 26866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:56.358289 26866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:56.365124 26866 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:36309
I20260810 18:15:56.365141 26988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:36309 every 8 connection(s)
I20260810 18:15:56.366039 26866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:56.368709 26990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:56.374006 26990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a: Bootstrap starting.
I20260810 18:15:56.375155 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 26866
I20260810 18:15:56.375306 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:56.375658 23813 external_mini_cluster.cc:1428] Setting key d0ca3df1dbc6348de2a534a5b8bcfd31
I20260810 18:15:56.376863 26990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:56.377578 26990 log.cc:824] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:56.379643 26990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a: No bootstrap required, opened a new log
I20260810 18:15:56.383671 26990 raft_consensus.cc:348] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1b44900a6f4bd4a0bff7ed92b4421a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36309 } }
I20260810 18:15:56.383898 26990 raft_consensus.cc:374] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:56.383960 26990 raft_consensus.cc:729] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1b44900a6f4bd4a0bff7ed92b4421a, State: Initialized, Role: FOLLOWER
I20260810 18:15:56.384336 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:56.376763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37449 (local address 127.23.65.126:36309)
0810 18:15:56.377158 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:56.377174 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:56.377208 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:15:56.377896 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:56.377922 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:56.378349 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:56.378636 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:56.378964 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:56.379971 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:56.381162 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:56.381615 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:56.381738 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:56.382557 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:56.382562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:56.382574 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:56.382966 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:56.383027 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:56.383051 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:56.383149 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:56.383330 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 18:15:56.383459 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260810 18:15:56.384351 26990 consensus_queue.cc:260] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1b44900a6f4bd4a0bff7ed92b4421a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36309 } }
I20260810 18:15:56.384552 26990 raft_consensus.cc:388] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:56.384609 26990 raft_consensus.cc:482] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:56.384709 26990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:56.385613 26990 raft_consensus.cc:504] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1b44900a6f4bd4a0bff7ed92b4421a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36309 } }
I20260810 18:15:56.385967 26990 leader_election.cc:302] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af1b44900a6f4bd4a0bff7ed92b4421a; no voters:
I20260810 18:15:56.386276 26990 leader_election.cc:288] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:56.386387 27000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:56.386612 27000 raft_consensus.cc:686] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [term 1 LEADER]: Becoming Leader. State: Replica: af1b44900a6f4bd4a0bff7ed92b4421a, State: Running, Role: LEADER
I20260810 18:15:56.386983 27000 consensus_queue.cc:237] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1b44900a6f4bd4a0bff7ed92b4421a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36309 } }
I20260810 18:15:56.387290 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:56.387487 26990 sys_catalog.cc:564] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:56.389655 27001 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af1b44900a6f4bd4a0bff7ed92b4421a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1b44900a6f4bd4a0bff7ed92b4421a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36309 } } }
I20260810 18:15:56.389777 27001 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:56.389969 27002 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [sys.catalog]: SysCatalogTable state changed. Reason: New leader af1b44900a6f4bd4a0bff7ed92b4421a. Latest consensus state: current_term: 1 leader_uuid: "af1b44900a6f4bd4a0bff7ed92b4421a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1b44900a6f4bd4a0bff7ed92b4421a" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36309 } } }
I20260810 18:15:56.390066 27002 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:56.390286 27038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:56.393352 27038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:56.393709 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:56.389732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41043 (local address 127.23.65.126:36309)
0810 18:15:56.389847 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:56.389851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:56.390386 (+ 535us) server_negotiation.cc:407] Connection header received
0810 18:15:56.390518 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:56.390520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:56.390557 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:56.390619 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:56.390898 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:56.391395 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:56.392216 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:56.392444 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:56.392528 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:56.393288 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:56.393291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:56.393294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:56.393306 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:56.393332 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:56.393342 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:56.393407 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:56.393553 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 18:15:56.393585 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:15:56.398571 27038 catalog_manager.cc:1347] Generated new cluster ID: b0f6b70d5ef446359ccf60781b4bb263
I20260810 18:15:56.398635 27038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:56.407966 27038 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:56.409098 27038 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:56.415390 27038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af1b44900a6f4bd4a0bff7ed92b4421a: Generated new TSK 0
I20260810 18:15:56.416211 27038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:56.425935 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 26866
W20260810 18:15:56.441242 26838 connection.cc:535] client connection to 127.23.65.126:36309 recv error: Network error: failed to read from TLS socket (remote: 127.23.65.126:36309): Connection reset by peer (error 104)
2026-08-10T18:15:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1513 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:56 dist-test-slave-jpj1 krb5kdc[27047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:56 dist-test-slave-jpj1 krb5kdc[27047](info): set up 2 sockets
Aug 10 18:15:56 dist-test-slave-jpj1 krb5kdc[27047](info): commencing operation
krb5kdc: starting...
W20260810 18:15:57.587218 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:57 dist-test-slave-jpj1 krb5kdc[27047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:15:57.714943 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:35393
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:57.827272 27063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:57.827553 27063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:57.827612 27063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:57.831516 27063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:57.831583 27063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:57.831601 27063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:57.831615 27063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:57.831629 27063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:57.836786 27063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:35393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:57.837738 27063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:57.838675 27063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:57.846753 27075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:57.846784 27071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:57.846784 27072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:57.847002 27063 server_base.cc:1034] running on GCE node
I20260810 18:15:57.847605 27063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:57.848672 27063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:57.849857 27063 hybrid_clock.cc:648] HybridClock initialized: now 1786385757849841 us; error 27 us; skew 500 ppm
Aug 10 18:15:57 dist-test-slave-jpj1 krb5kdc[27047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385757, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:57.852563 27063 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:15:57.853461 27063 webserver.cc:460] Webserver started at http://127.23.65.126:39845/ using document root <none> and password file <none>
I20260810 18:15:57.853987 27063 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:57.854056 27063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:57.854243 27063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:57.856315 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "f4d422032bc448ac9bcf9f984c358b24"
format_stamp: "Formatted at 2026-08-10 18:15:57 on dist-test-slave-jpj1"
server_key: "5e633263ba288a5bf45841cc75d8a46e"
server_key_iv: "e82b7e5369cde2ee5874faf9d34ae4f2"
server_key_version: "encryptionkey@0"
I20260810 18:15:57.856853 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "f4d422032bc448ac9bcf9f984c358b24"
format_stamp: "Formatted at 2026-08-10 18:15:57 on dist-test-slave-jpj1"
server_key: "5e633263ba288a5bf45841cc75d8a46e"
server_key_iv: "e82b7e5369cde2ee5874faf9d34ae4f2"
server_key_version: "encryptionkey@0"
I20260810 18:15:57.860598 27063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:15:57.862887 27084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:57.864076 27063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:57.864200 27063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "f4d422032bc448ac9bcf9f984c358b24"
format_stamp: "Formatted at 2026-08-10 18:15:57 on dist-test-slave-jpj1"
server_key: "5e633263ba288a5bf45841cc75d8a46e"
server_key_iv: "e82b7e5369cde2ee5874faf9d34ae4f2"
server_key_version: "encryptionkey@0"
I20260810 18:15:57.864315 27063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:57.868813 27063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:57.871990 27063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:57.872152 27063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:57.878834 27063 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:35393
I20260810 18:15:57.878850 27172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:35393 every 8 connection(s)
I20260810 18:15:57.879823 27063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:57.881688 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 27063
I20260810 18:15:57.881846 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:57.882108 23813 external_mini_cluster.cc:1428] Setting key 744918499002a071de726be65ff28e44
I20260810 18:15:57.882690 27173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:57.887581 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:57.883424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45643 (local address 127.23.65.126:35393)
0810 18:15:57.883887 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:57.883906 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:57.883949 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:15:57.884796 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:57.884821 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:57.885191 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:57.885555 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:57.886191 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:57.886196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:57.886210 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:57.886222 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:57.886282 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:57.886307 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:57.886448 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:57.886520 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 18:15:57.886672 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":165,"threads_started":1}
I20260810 18:15:57.889667 27173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24: Bootstrap starting.
I20260810 18:15:57.890456 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:57.891974 27173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:57.892835 27173 log.cc:824] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:57.893944 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:57.892429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36837 (local address 127.23.65.126:35393)
0810 18:15:57.892530 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:57.892534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:57.892752 (+ 218us) server_negotiation.cc:407] Connection header received
0810 18:15:57.892867 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:57.892870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:57.892901 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:57.892961 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:57.893598 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:57.893601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:57.893604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:15:57.893616 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:57.893643 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:15:57.893653 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:57.893721 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:57.893799 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 18:15:57.893835 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 18:15:57.894858 27173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24: No bootstrap required, opened a new log
I20260810 18:15:57.898315 27173 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d422032bc448ac9bcf9f984c358b24" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35393 } }
I20260810 18:15:57.898531 27173 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:57.898595 27173 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d422032bc448ac9bcf9f984c358b24, State: Initialized, Role: FOLLOWER
I20260810 18:15:57.898986 27173 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d422032bc448ac9bcf9f984c358b24" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35393 } }
I20260810 18:15:57.899118 27173 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:57.899185 27173 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:57.899290 27173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:57.900274 27173 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d422032bc448ac9bcf9f984c358b24" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35393 } }
I20260810 18:15:57.900592 27173 leader_election.cc:302] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4d422032bc448ac9bcf9f984c358b24; no voters:
I20260810 18:15:57.900875 27173 leader_election.cc:288] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:57.900977 27189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:57.901212 27189 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [term 1 LEADER]: Becoming Leader. State: Replica: f4d422032bc448ac9bcf9f984c358b24, State: Running, Role: LEADER
I20260810 18:15:57.901511 27189 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d422032bc448ac9bcf9f984c358b24" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35393 } }
I20260810 18:15:57.902037 27173 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:57.903053 27191 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4d422032bc448ac9bcf9f984c358b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d422032bc448ac9bcf9f984c358b24" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35393 } } }
I20260810 18:15:57.903196 27191 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:57.903474 27192 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4d422032bc448ac9bcf9f984c358b24. Latest consensus state: current_term: 1 leader_uuid: "f4d422032bc448ac9bcf9f984c358b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d422032bc448ac9bcf9f984c358b24" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35393 } } }
I20260810 18:15:57.903600 27192 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:57.903712 27203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:57.906833 27203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:57.912465 27203 catalog_manager.cc:1347] Generated new cluster ID: 89bc09f3381c4805ab95a4ce4414d818
I20260810 18:15:57.912530 27203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:57.924842 27203 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:57.925197 27203 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:57.933172 27203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4d422032bc448ac9bcf9f984c358b24: Generated new TSK 0
I20260810 18:15:57.933900 27203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:57.938271 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 27063
2026-08-10T18:15:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1509 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:57 dist-test-slave-jpj1 krb5kdc[27227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:57 dist-test-slave-jpj1 krb5kdc[27227](info): set up 2 sockets
Aug 10 18:15:57 dist-test-slave-jpj1 krb5kdc[27227](info): commencing operation
krb5kdc: starting...
W20260810 18:15:59.099521 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:15:59 dist-test-slave-jpj1 krb5kdc[27227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:15:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:15:59.226145 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:35817
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:35817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:15:59.338598 27250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:15:59.338876 27250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:15:59.338934 27250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:15:59.342909 27250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:15:59.342974 27250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:15:59.343000 27250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:15:59.343020 27250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:15:59.343041 27250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:15:59.348232 27250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:35817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:35817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:15:59.349231 27250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:15:59.350224 27250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:15:59.358284 27262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:59.358292 27261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:15:59.358299 27264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:15:59.358882 27250 server_base.cc:1034] running on GCE node
I20260810 18:15:59.359439 27250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:15:59.360536 27250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:15:59.361702 27250 hybrid_clock.cc:648] HybridClock initialized: now 1786385759361685 us; error 26 us; skew 500 ppm
Aug 10 18:15:59 dist-test-slave-jpj1 krb5kdc[27227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385759, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:15:59.364472 27250 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:15:59.365391 27250 webserver.cc:460] Webserver started at http://127.23.65.126:35239/ using document root <none> and password file <none>
I20260810 18:15:59.365926 27250 fs_manager.cc:359] Metadata directory not provided
I20260810 18:15:59.365994 27250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:15:59.366237 27250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:15:59.368287 27250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "5243de021331432bbd6b328ad34f4901"
format_stamp: "Formatted at 2026-08-10 18:15:59 on dist-test-slave-jpj1"
server_key: "acd15f2bbb56d7e4fa59b98c99560bb2"
server_key_iv: "50ff0d9a2637fbd6c00fc7616e06cded"
server_key_version: "encryptionkey@0"
I20260810 18:15:59.368830 27250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "5243de021331432bbd6b328ad34f4901"
format_stamp: "Formatted at 2026-08-10 18:15:59 on dist-test-slave-jpj1"
server_key: "acd15f2bbb56d7e4fa59b98c99560bb2"
server_key_iv: "50ff0d9a2637fbd6c00fc7616e06cded"
server_key_version: "encryptionkey@0"
I20260810 18:15:59.372520 27250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:15:59.374933 27274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:15:59.376179 27250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:15:59.376319 27250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "5243de021331432bbd6b328ad34f4901"
format_stamp: "Formatted at 2026-08-10 18:15:59 on dist-test-slave-jpj1"
server_key: "acd15f2bbb56d7e4fa59b98c99560bb2"
server_key_iv: "50ff0d9a2637fbd6c00fc7616e06cded"
server_key_version: "encryptionkey@0"
I20260810 18:15:59.376421 27250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:15:59.401520 27250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:15:59.405035 27250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:15:59.405243 27250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:15:59.413578 27250 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:35817
I20260810 18:15:59.413610 27365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:35817 every 8 connection(s)
I20260810 18:15:59.414535 27250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:15:59.417409 27366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:15:59.422891 27366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901: Bootstrap starting.
I20260810 18:15:59.423424 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 27250
I20260810 18:15:59.423568 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:15:59.423859 23813 external_mini_cluster.cc:1428] Setting key 86fb7501917cfdced07393a6b37c2198
I20260810 18:15:59.425567 27366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901: Neither blocks nor log segments found. Creating new log.
I20260810 18:15:59.426303 27366 log.cc:824] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901: Log is configured to *not* fsync() on all Append() calls
I20260810 18:15:59.428459 27366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901: No bootstrap required, opened a new log
Aug 10 18:15:59 dist-test-slave-jpj1 krb5kdc[27227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:15:59.432659 27366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5243de021331432bbd6b328ad34f4901" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35817 } }
I20260810 18:15:59.432871 27366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:15:59.432927 27366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5243de021331432bbd6b328ad34f4901, State: Initialized, Role: FOLLOWER
I20260810 18:15:59.433353 27366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5243de021331432bbd6b328ad34f4901" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35817 } }
I20260810 18:15:59.433521 27366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:15:59.433593 27366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:15:59.433710 27366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:15:59.434798 27366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5243de021331432bbd6b328ad34f4901" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35817 } }
I20260810 18:15:59.435158 27366 leader_election.cc:302] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5243de021331432bbd6b328ad34f4901; no voters:
I20260810 18:15:59.435518 27366 leader_election.cc:288] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:15:59.435650 27375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:15:59.435884 27375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [term 1 LEADER]: Becoming Leader. State: Replica: 5243de021331432bbd6b328ad34f4901, State: Running, Role: LEADER
I20260810 18:15:59.436260 27375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5243de021331432bbd6b328ad34f4901" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35817 } }
I20260810 18:15:59.436841 27366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:15:59.437986 27378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5243de021331432bbd6b328ad34f4901. Latest consensus state: current_term: 1 leader_uuid: "5243de021331432bbd6b328ad34f4901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5243de021331432bbd6b328ad34f4901" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35817 } } }
I20260810 18:15:59.438100 27378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:59.438436 27376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5243de021331432bbd6b328ad34f4901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5243de021331432bbd6b328ad34f4901" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35817 } } }
I20260810 18:15:59.438495 27386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:15:59.438570 27376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901 [sys.catalog]: This master's current role is: LEADER
I20260810 18:15:59.440565 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:59.425067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59317 (local address 127.23.65.126:35817)
0810 18:15:59.425493 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:59.425509 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:59.425544 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:15:59.426220 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:59.426238 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:59.426607 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:59.426973 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:59.427851 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:59.428863 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:59.430031 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:59.430498 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:59.430609 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:59.433384 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:59.433390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:59.433405 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:59.433438 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:59.435317 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:59.435857 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:59.435863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:59.435869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:59.436053 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:59.436342 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:59.436345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:59.436347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:59.436850 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:59.437041 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:59.437504 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 18:15:59.437829 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20260810 18:15:59.441810 27386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:15:59.443584 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:15:59.447860 27386 catalog_manager.cc:1347] Generated new cluster ID: 351dd68b676544fbab8e8994eaf54de1
I20260810 18:15:59.447918 27386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:15:59.454159 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:15:59.445457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48473 (local address 127.23.65.126:35817)
0810 18:15:59.445541 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 18:15:59.445545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:15:59.445811 (+ 266us) server_negotiation.cc:407] Connection header received
0810 18:15:59.445971 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:15:59.445974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:15:59.446017 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:15:59.446083 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:15:59.446962 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:59.447767 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:59.448931 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:15:59.449291 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:15:59.449385 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:15:59.450939 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:15:59.450943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:15:59.450946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:15:59.450969 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:15:59.452500 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:59.452965 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:59.452968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:59.452970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:59.453047 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:15:59.453288 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:15:59.453290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:15:59.453292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:15:59.453467 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:15:59.453587 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:15:59.453784 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 18:15:59.453887 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 18:15:59.466050 27386 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:15:59.467258 27386 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:15:59.474953 27386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5243de021331432bbd6b328ad34f4901: Generated new TSK 0
I20260810 18:15:59.475749 27386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:15:59.481045 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 27250
2026-08-10T18:15:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1549 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:15:59 dist-test-slave-jpj1 krb5kdc[27427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:15:59 dist-test-slave-jpj1 krb5kdc[27427](info): set up 2 sockets
Aug 10 18:15:59 dist-test-slave-jpj1 krb5kdc[27427](info): commencing operation
krb5kdc: starting...
W20260810 18:16:00.650527 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:16:00 dist-test-slave-jpj1 krb5kdc[27427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:16:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:16:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:16:00.774261 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:36561
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:36561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:16:00.886215 27446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:16:00.886471 27446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:16:00.886516 27446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:16:00.890497 27446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:16:00.890553 27446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:16:00.890571 27446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:16:00.890588 27446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:16:00.890618 27446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:16:00.895758 27446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:36561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:36561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:16:00.896708 27446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:16:00.897650 27446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:16:00.905664 27457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:00.905676 27456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:00.905679 27461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:00.905906 27446 server_base.cc:1034] running on GCE node
I20260810 18:16:00.906545 27446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:16:00.907588 27446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:16:00.908753 27446 hybrid_clock.cc:648] HybridClock initialized: now 1786385760908739 us; error 27 us; skew 500 ppm
Aug 10 18:16:00 dist-test-slave-jpj1 krb5kdc[27427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385760, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:16:00.911278 27446 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:16:00.912117 27446 webserver.cc:460] Webserver started at http://127.23.65.126:43475/ using document root <none> and password file <none>
I20260810 18:16:00.912597 27446 fs_manager.cc:359] Metadata directory not provided
I20260810 18:16:00.912652 27446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:16:00.912825 27446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:16:00.914757 27446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "cf2290c752704ab48a59e62aa22564de"
format_stamp: "Formatted at 2026-08-10 18:16:00 on dist-test-slave-jpj1"
server_key: "638a3ea5d315342e89084fb538da95bc"
server_key_iv: "92d0bcf7be49245179dfaf596e7cfb45"
server_key_version: "encryptionkey@0"
I20260810 18:16:00.915273 27446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "cf2290c752704ab48a59e62aa22564de"
format_stamp: "Formatted at 2026-08-10 18:16:00 on dist-test-slave-jpj1"
server_key: "638a3ea5d315342e89084fb538da95bc"
server_key_iv: "92d0bcf7be49245179dfaf596e7cfb45"
server_key_version: "encryptionkey@0"
I20260810 18:16:00.918716 27446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:16:00.920971 27476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:16:00.922129 27446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:16:00.922237 27446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "cf2290c752704ab48a59e62aa22564de"
format_stamp: "Formatted at 2026-08-10 18:16:00 on dist-test-slave-jpj1"
server_key: "638a3ea5d315342e89084fb538da95bc"
server_key_iv: "92d0bcf7be49245179dfaf596e7cfb45"
server_key_version: "encryptionkey@0"
I20260810 18:16:00.922326 27446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:16:00.946951 27446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:16:00.949923 27446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:16:00.950086 27446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:16:00.956954 27446 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:36561
I20260810 18:16:00.956971 27555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:36561 every 8 connection(s)
I20260810 18:16:00.957957 27446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:16:00.960595 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 27446
I20260810 18:16:00.960749 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:16:00.961059 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:16:00.961090 23813 external_mini_cluster.cc:1428] Setting key 49a0148ff93f1e04a322659f12f0bf96
I20260810 18:16:00.967418 27558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de: Bootstrap starting.
I20260810 18:16:00.969931 27558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de: Neither blocks nor log segments found. Creating new log.
I20260810 18:16:00.970047 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:00.962553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33127 (local address 127.23.65.126:36561)
0810 18:16:00.962992 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:00.963007 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:00.963044 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:16:00.963843 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:00.963862 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:00.964201 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:00.964512 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:00.964918 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:00.965900 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:00.966773 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:00.967226 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:00.967380 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:00.968170 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:00.968175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:00.968187 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:16:00.968561 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:00.968615 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:16:00.968637 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:00.968722 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:00.968862 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 18:16:00.969078 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":148,"threads_started":1}
I20260810 18:16:00.970815 27558 log.cc:824] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de: Log is configured to *not* fsync() on all Append() calls
I20260810 18:16:00.972744 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:16:00.972961 27558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de: No bootstrap required, opened a new log
I20260810 18:16:00.977020 27558 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2290c752704ab48a59e62aa22564de" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36561 } }
I20260810 18:16:00.977316 27558 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:16:00.977382 27558 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2290c752704ab48a59e62aa22564de, State: Initialized, Role: FOLLOWER
I20260810 18:16:00.977821 27558 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2290c752704ab48a59e62aa22564de" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36561 } }
I20260810 18:16:00.977962 27558 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:16:00.978027 27558 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:16:00.978121 27558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:16:00.978453 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:00.974573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44349 (local address 127.23.65.126:36561)
0810 18:16:00.974692 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:00.974696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:00.974907 (+ 211us) server_negotiation.cc:407] Connection header received
0810 18:16:00.974981 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:00.974984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:00.975030 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:00.975112 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:00.975316 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:00.976125 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:00.976934 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:00.977237 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:00.977353 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:00.978071 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:00.978075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:00.978077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:16:00.978091 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:00.978117 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:16:00.978129 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:00.978188 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:00.978277 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 18:16:00.978328 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:16:00.979211 27558 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2290c752704ab48a59e62aa22564de" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36561 } }
I20260810 18:16:00.979584 27558 leader_election.cc:302] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf2290c752704ab48a59e62aa22564de; no voters:
I20260810 18:16:00.979908 27558 leader_election.cc:288] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:16:00.980070 27580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:16:00.980291 27580 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [term 1 LEADER]: Becoming Leader. State: Replica: cf2290c752704ab48a59e62aa22564de, State: Running, Role: LEADER
I20260810 18:16:00.980621 27580 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2290c752704ab48a59e62aa22564de" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36561 } }
I20260810 18:16:00.981104 27558 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:16:00.982115 27585 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf2290c752704ab48a59e62aa22564de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2290c752704ab48a59e62aa22564de" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36561 } } }
I20260810 18:16:00.982239 27585 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [sys.catalog]: This master's current role is: LEADER
I20260810 18:16:00.982630 27596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:16:00.982818 27586 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2290c752704ab48a59e62aa22564de. Latest consensus state: current_term: 1 leader_uuid: "cf2290c752704ab48a59e62aa22564de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2290c752704ab48a59e62aa22564de" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 36561 } } }
I20260810 18:16:00.983393 27586 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de [sys.catalog]: This master's current role is: LEADER
I20260810 18:16:00.985669 27596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:16:00.991078 27596 catalog_manager.cc:1347] Generated new cluster ID: 4a79091f7a6542fea22430a0c540c25a
I20260810 18:16:00.991135 27596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:16:01.014530 27596 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:16:01.015594 27596 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:16:01.025254 27596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf2290c752704ab48a59e62aa22564de: Generated new TSK 0
I20260810 18:16:01.026023 27596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:16:01.040402 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 27446
2026-08-10T18:16:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1551 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:16:01 dist-test-slave-jpj1 krb5kdc[27614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:16:01 dist-test-slave-jpj1 krb5kdc[27614](info): set up 2 sockets
Aug 10 18:16:01 dist-test-slave-jpj1 krb5kdc[27614](info): commencing operation
krb5kdc: starting...
W20260810 18:16:02.208398 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:16:02 dist-test-slave-jpj1 krb5kdc[27614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:16:02.330480 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:35303
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:35303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:16:02.439620 27635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:16:02.439893 27635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:16:02.439952 27635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:16:02.443888 27635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:16:02.443953 27635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:16:02.443979 27635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:16:02.444000 27635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:16:02.444027 27635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:16:02.449332 27635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:35303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:35303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:16:02.450373 27635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:16:02.451364 27635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:16:02.459810 27635 server_base.cc:1034] running on GCE node
W20260810 18:16:02.459859 27652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:02.459872 27655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:02.459966 27659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:02.460659 27635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:16:02.461752 27635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:16:02.462924 27635 hybrid_clock.cc:648] HybridClock initialized: now 1786385762462905 us; error 30 us; skew 500 ppm
Aug 10 18:16:02 dist-test-slave-jpj1 krb5kdc[27614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:16:02.465667 27635 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:16:02.466578 27635 webserver.cc:460] Webserver started at http://127.23.65.126:39037/ using document root <none> and password file <none>
I20260810 18:16:02.467111 27635 fs_manager.cc:359] Metadata directory not provided
I20260810 18:16:02.467176 27635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:16:02.467371 27635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:16:02.469424 27635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f"
format_stamp: "Formatted at 2026-08-10 18:16:02 on dist-test-slave-jpj1"
server_key: "788ed72ca7dbb5df131efe24c43c7059"
server_key_iv: "c18d0d69799c4d71f87726f0ee65bbb9"
server_key_version: "encryptionkey@0"
I20260810 18:16:02.469978 27635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f"
format_stamp: "Formatted at 2026-08-10 18:16:02 on dist-test-slave-jpj1"
server_key: "788ed72ca7dbb5df131efe24c43c7059"
server_key_iv: "c18d0d69799c4d71f87726f0ee65bbb9"
server_key_version: "encryptionkey@0"
I20260810 18:16:02.473574 27635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:16:02.475857 27671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:16:02.477059 27635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:16:02.477183 27635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f"
format_stamp: "Formatted at 2026-08-10 18:16:02 on dist-test-slave-jpj1"
server_key: "788ed72ca7dbb5df131efe24c43c7059"
server_key_iv: "c18d0d69799c4d71f87726f0ee65bbb9"
server_key_version: "encryptionkey@0"
I20260810 18:16:02.477277 27635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:16:02.481732 27635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:16:02.484381 27635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:16:02.484547 27635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:16:02.491285 27635 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:35303
I20260810 18:16:02.491298 27767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:35303 every 8 connection(s)
I20260810 18:16:02.492219 27635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:16:02.494885 27768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:16:02.496634 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 27635
I20260810 18:16:02.496778 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:16:02.497073 23813 external_mini_cluster.cc:1428] Setting key 52a4fd068df19ff53934d40eee165a73
I20260810 18:16:02.501425 27768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f: Bootstrap starting.
I20260810 18:16:02.502499 27774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:02.498528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50145 (local address 127.23.65.126:35303)
0810 18:16:02.498964 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:02.498979 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:02.499014 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:16:02.499713 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:02.499732 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:02.500005 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:02.500282 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:02.501185 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:02.501190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:02.501203 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:16:02.501218 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:02.501275 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:16:02.501294 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:02.501408 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:02.501541 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 18:16:02.501835 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":134,"threads_started":1}
I20260810 18:16:02.503705 27768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f: Neither blocks nor log segments found. Creating new log.
I20260810 18:16:02.504520 27768 log.cc:824] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:16:02.505340 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:16:02.506647 27768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f: No bootstrap required, opened a new log
I20260810 18:16:02.509128 27774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:02.507614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44965 (local address 127.23.65.126:35303)
0810 18:16:02.507734 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:02.507738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:02.507950 (+ 212us) server_negotiation.cc:407] Connection header received
0810 18:16:02.508047 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:02.508050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:02.508080 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:02.508144 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:02.508790 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:02.508794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:02.508796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:16:02.508807 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:02.508834 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:16:02.508844 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:02.508912 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:02.508984 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 18:16:02.509019 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:16:02.510480 27768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35303 } }
I20260810 18:16:02.510691 27768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:16:02.510733 27768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662cffa9eeb64fdb89ac1d4ec0034b4f, State: Initialized, Role: FOLLOWER
I20260810 18:16:02.511113 27768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35303 } }
I20260810 18:16:02.511248 27768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:16:02.511297 27768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:16:02.511368 27768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:16:02.512200 27768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35303 } }
I20260810 18:16:02.512526 27768 leader_election.cc:302] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 662cffa9eeb64fdb89ac1d4ec0034b4f; no voters:
I20260810 18:16:02.512809 27768 leader_election.cc:288] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:16:02.512892 27788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:16:02.513062 27788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [term 1 LEADER]: Becoming Leader. State: Replica: 662cffa9eeb64fdb89ac1d4ec0034b4f, State: Running, Role: LEADER
I20260810 18:16:02.513356 27788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35303 } }
I20260810 18:16:02.514066 27768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:16:02.515030 27793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 662cffa9eeb64fdb89ac1d4ec0034b4f. Latest consensus state: current_term: 1 leader_uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35303 } } }
I20260810 18:16:02.515161 27793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [sys.catalog]: This master's current role is: LEADER
I20260810 18:16:02.515645 27803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:16:02.515642 27789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662cffa9eeb64fdb89ac1d4ec0034b4f" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 35303 } } }
I20260810 18:16:02.516113 27789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f [sys.catalog]: This master's current role is: LEADER
I20260810 18:16:02.518718 27803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:16:02.523772 27803 catalog_manager.cc:1347] Generated new cluster ID: 7876e04f0f92465c9a111d2d8be6c0c2
I20260810 18:16:02.523838 27803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:16:02.534843 27803 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:16:02.535130 27803 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:16:02.542028 27803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 662cffa9eeb64fdb89ac1d4ec0034b4f: Generated new TSK 0
I20260810 18:16:02.542651 27803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:16:02.550614 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 27635
2026-08-10T18:16:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1512 ms)
[----------] 18 tests from AuthTokenIssuingTest (27616 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:16:02 dist-test-slave-jpj1 krb5kdc[27827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:16:02 dist-test-slave-jpj1 krb5kdc[27827](info): set up 2 sockets
Aug 10 18:16:02 dist-test-slave-jpj1 krb5kdc[27827](info): commencing operation
krb5kdc: starting...
W20260810 18:16:03.704294 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:16:03 dist-test-slave-jpj1 krb5kdc[27827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:16:03.826262 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:37137
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:37137,127.23.65.125:42021,127.23.65.124:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:16:03.940142 27846 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:16:03.940413 27846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:16:03.940434 27846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:16:03.940469 27846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:16:03.944335 27846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:16:03.944386 27846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:16:03.944403 27846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:16:03.944417 27846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:16:03.944430 27846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:16:03.949504 27846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:37137,127.23.65.125:42021,127.23.65.124:38825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:37137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:16:03.950414 27846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:16:03.951326 27846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:16:03.959316 27856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:03.959374 27846 server_base.cc:1034] running on GCE node
W20260810 18:16:03.959326 27854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:03.959328 27862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:03.960207 27846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:16:03.961220 27846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:16:03.962369 27846 hybrid_clock.cc:648] HybridClock initialized: now 1786385763962346 us; error 36 us; skew 500 ppm
Aug 10 18:16:03 dist-test-slave-jpj1 krb5kdc[27827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:16:03.964900 27846 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:16:03.965721 27846 webserver.cc:460] Webserver started at http://127.23.65.126:34331/ using document root <none> and password file <none>
I20260810 18:16:03.966223 27846 fs_manager.cc:359] Metadata directory not provided
I20260810 18:16:03.966293 27846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:16:03.966493 27846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:16:03.968488 27846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "9b3f1b7eadd146398b8a3dfe85f1a588"
format_stamp: "Formatted at 2026-08-10 18:16:03 on dist-test-slave-jpj1"
server_key: "658932061965ee4f1205994cba846e5b"
server_key_iv: "a768b60b39960e67cfa8caba29826ef7"
server_key_version: "encryptionkey@0"
I20260810 18:16:03.969003 27846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "9b3f1b7eadd146398b8a3dfe85f1a588"
format_stamp: "Formatted at 2026-08-10 18:16:03 on dist-test-slave-jpj1"
server_key: "658932061965ee4f1205994cba846e5b"
server_key_iv: "a768b60b39960e67cfa8caba29826ef7"
server_key_version: "encryptionkey@0"
I20260810 18:16:03.972462 27846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:16:03.974654 27870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:16:03.975834 27846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:16:03.975951 27846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "9b3f1b7eadd146398b8a3dfe85f1a588"
format_stamp: "Formatted at 2026-08-10 18:16:03 on dist-test-slave-jpj1"
server_key: "658932061965ee4f1205994cba846e5b"
server_key_iv: "a768b60b39960e67cfa8caba29826ef7"
server_key_version: "encryptionkey@0"
I20260810 18:16:03.976039 27846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:16:03.990171 27846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:16:03.992856 27846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:16:03.993019 27846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:16:03.999703 27846 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:37137
I20260810 18:16:03.999967 27952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:37137 every 8 connection(s)
I20260810 18:16:04.000631 27846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:16:04.001932 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 27846
I20260810 18:16:04.002084 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:16:04.002399 23813 external_mini_cluster.cc:1428] Setting key 4fa3182c334fc465382fb36690ae4471
I20260810 18:16:04.003849 27955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:16:04.008505 27955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.65.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.125@KRBTEST.COM" created.
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:16:04.024391 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.011348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:37137 (local address 127.0.0.1:57460)
0810 18:16:04.012007 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.012056 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:04.012891 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:04.013574 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:04.013579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:04.013903 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:04.014183 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.014202 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.015560 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.015563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.016640 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.016652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.017488 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.017491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.017585 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.018138 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:04.018154 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:04.019742 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:04.021529 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.021540 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.021551 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.021864 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.022156 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.022159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.022161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.022325 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.022897 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:04.022904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:04.023085 (+ 181us) client_negotiation.cc:769] Sending connection context
0810 18:16:04.023279 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 18:16:04.023677 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":86,"spinlock_wait_cycles":42752,"thread_start_us":185,"threads_started":1}
I20260810 18:16:04.024396 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.011348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.23.65.126:37137)
0810 18:16:04.011886 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.011909 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.012172 (+ 263us) server_negotiation.cc:407] Connection header received
0810 18:16:04.013188 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.013193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.013390 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.013546 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.014317 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.015407 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.016788 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.017342 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.017504 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.019890 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.019895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.019906 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.019927 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.021417 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.021965 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.021969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.021971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.022070 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.022413 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.022415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.022416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.022776 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.022879 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.023332 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.023552 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":179,"threads_started":1}
I20260810 18:16:04.026736 27955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:04.027444 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.027184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:42021 (local address 127.0.0.1:36752)
0810 18:16:04.027316 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.027385 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:16:04.027676 27872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
I20260810 18:16:04.028508 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.028313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:42021 (local address 127.0.0.1:36762)
0810 18:16:04.028412 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.028457 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 18:16:04.028641 27955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.125:42021: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab.
Entry for principal kudu/127.23.65.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab.
I20260810 18:16:04.030300 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.125:42021
--webserver_interface=127.23.65.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab
--principal=kudu/127.23.65.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:37137,127.23.65.125:42021,127.23.65.124:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:16:04.077868 27955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } attempt: 1
I20260810 18:16:04.078765 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.078450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:42021 (local address 127.0.0.1:36778)
0810 18:16:04.078597 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.078662 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 18:16:04.079460 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.079222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:42021 (local address 127.0.0.1:36784)
0810 18:16:04.079333 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.079389 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 18:16:04.079603 27955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.125:42021: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
W20260810 18:16:04.141777 27971 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:16:04.142071 27971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:16:04.142105 27971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:16:04.142155 27971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:16:04.146129 27971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:16:04.146185 27971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:16:04.146214 27971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:16:04.146235 27971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:16:04.146262 27971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:16:04.147749 27955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } attempt: 2
I20260810 18:16:04.148628 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.148302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:42021 (local address 127.0.0.1:36800)
0810 18:16:04.148446 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.148533 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 18:16:04.149330 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.149102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:42021 (local address 127.0.0.1:36806)
0810 18:16:04.149207 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.149270 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 18:16:04.149437 27955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.125:42021: Network error: Client connection negotiation failed: client connection to 127.23.65.125:42021: connect: Connection refused (error 111)
I20260810 18:16:04.151932 27971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:37137,127.23.65.125:42021,127.23.65.124:38825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab
--principal=kudu/127.23.65.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.125:42021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.65.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:16:04.152956 27971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:16:04.153961 27971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:16:04.162212 27977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:04.162364 27971 server_base.cc:1034] running on GCE node
W20260810 18:16:04.162243 27981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:04.162212 27979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:04.162963 27971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:16:04.164062 27971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:16:04.165242 27971 hybrid_clock.cc:648] HybridClock initialized: now 1786385764165218 us; error 34 us; skew 500 ppm
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:16:04.167900 27971 init.cc:377] Logged in from keytab as kudu/127.23.65.125@KRBTEST.COM (short username kudu)
I20260810 18:16:04.168741 27971 webserver.cc:460] Webserver started at http://127.23.65.125:46149/ using document root <none> and password file <none>
I20260810 18:16:04.169258 27971 fs_manager.cc:359] Metadata directory not provided
I20260810 18:16:04.169324 27971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:16:04.169528 27971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:16:04.171499 27971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/data/instance:
uuid: "6fba6cd4de454857b12f03b21b88bc04"
format_stamp: "Formatted at 2026-08-10 18:16:04 on dist-test-slave-jpj1"
server_key: "257e290edb6d7e20079bb281bf25d502"
server_key_iv: "290da64e55cc3ba9f15df494dd2abffa"
server_key_version: "encryptionkey@0"
I20260810 18:16:04.172058 27971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/wal/instance:
uuid: "6fba6cd4de454857b12f03b21b88bc04"
format_stamp: "Formatted at 2026-08-10 18:16:04 on dist-test-slave-jpj1"
server_key: "257e290edb6d7e20079bb281bf25d502"
server_key_iv: "290da64e55cc3ba9f15df494dd2abffa"
server_key_version: "encryptionkey@0"
I20260810 18:16:04.175590 27971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:16:04.177856 27993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:16:04.179034 27971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:16:04.179143 27971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/wal
uuid: "6fba6cd4de454857b12f03b21b88bc04"
format_stamp: "Formatted at 2026-08-10 18:16:04 on dist-test-slave-jpj1"
server_key: "257e290edb6d7e20079bb281bf25d502"
server_key_iv: "290da64e55cc3ba9f15df494dd2abffa"
server_key_version: "encryptionkey@0"
I20260810 18:16:04.179236 27971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:16:04.194041 27971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:16:04.197070 27971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:16:04.197247 27971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:16:04.203933 27971 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.125:42021
I20260810 18:16:04.204171 28073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.125:42021 every 8 connection(s)
I20260810 18:16:04.204856 27971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
I20260810 18:16:04.206641 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 27971
I20260810 18:16:04.206784 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-1/wal/instance
I20260810 18:16:04.207077 23813 external_mini_cluster.cc:1428] Setting key 0f540324f147540a2db198ab950fff28
I20260810 18:16:04.207908 28075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:16:04.212335 28075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.65.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.124@KRBTEST.COM" created.
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:16:04.225443 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.214196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.23.65.126:37137)
0810 18:16:04.214326 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.214330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.215359 (+ 1029us) server_negotiation.cc:407] Connection header received
0810 18:16:04.216312 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.216315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.216375 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.216451 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.217648 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.218179 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.219364 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.219639 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.219732 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.222049 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.222052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.222054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.222083 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.223368 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.223910 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.223914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.223915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.223993 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.224321 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.224324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.224326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.224514 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.224602 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.225131 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.225268 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:16:04.226234 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.214697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:37137 (local address 127.0.0.1:57474)
0810 18:16:04.215193 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.215235 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:04.216102 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:04.216629 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:04.216637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:04.217007 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:04.217499 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.217513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.218316 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.218319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.219226 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.219238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.219799 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.219804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.219967 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.220510 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:04.220527 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:04.221883 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:04.223480 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.223486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.223501 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.223815 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.224078 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.224081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.224083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.224225 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.224616 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:04.224622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:04.224882 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 18:16:04.225051 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 18:16:04.225392 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
I20260810 18:16:04.227694 28075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab.
Entry for principal kudu/127.23.65.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab.
I20260810 18:16:04.232831 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.124:38825
--webserver_interface=127.23.65.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab
--principal=kudu/127.23.65.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:37137,127.23.65.125:42021,127.23.65.124:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
I20260810 18:16:04.236713 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.228215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:42021 (local address 127.0.0.1:36822)
0810 18:16:04.228607 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.228619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:04.228681 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:04.229025 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:04.229027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:04.229159 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:04.229259 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.229265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.230290 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.230293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.230885 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.230894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.231496 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.231499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.231603 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.232069 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:04.232081 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:04.233364 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:04.235193 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.235197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.235199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.235430 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.235733 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.235737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.235738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.235841 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.236238 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:04.236241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:04.236337 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 18:16:04.236434 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 18:16:04.236551 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334}
I20260810 18:16:04.236776 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.228324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.23.65.125:42021)
0810 18:16:04.228623 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.228633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.228655 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:16:04.228750 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.228754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.228917 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.229044 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.229354 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.230200 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.230982 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.231398 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.231482 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.233495 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.233506 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.233520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.233546 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.235082 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.235521 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.235525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.235527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.235634 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.235918 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.235921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.235922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.236143 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.236242 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.236508 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.236623 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":123,"threads_started":1}
I20260810 18:16:04.238487 28075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:04.239190 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.238920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56850)
0810 18:16:04.239036 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.239121 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 18:16:04.239392 27999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
I20260810 18:16:04.240267 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.240042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56862)
0810 18:16:04.240153 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.240205 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 18:16:04.240393 28075 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:38825: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
I20260810 18:16:04.240569 27955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } attempt: 3
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
I20260810 18:16:04.248392 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.241001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.23.65.125:42021)
0810 18:16:04.241098 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.241102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.241142 (+ 40us) server_negotiation.cc:407] Connection header received
0810 18:16:04.241275 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.241277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.241310 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.241366 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.241825 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.242316 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.243347 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.243596 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.243674 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.245717 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.245720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.245722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.245743 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.246943 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.247346 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.247348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.247349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.247412 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.247693 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.247695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.247696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.247859 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.247932 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.248199 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.248288 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:16:04.248430 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.240942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:42021 (local address 127.0.0.1:36828)
0810 18:16:04.241074 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.241096 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:04.241194 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:04.241408 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:04.241412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:04.241593 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:04.241716 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.241723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.242415 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.242419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.243242 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.243254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.243685 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.243689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.243815 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.244229 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:04.244244 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:04.245588 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:04.247031 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.247034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.247036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.247268 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.247479 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.247482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.247484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.247618 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.247935 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:04.247938 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:04.248038 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 18:16:04.248139 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 18:16:04.248265 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 18:16:04.249738 27955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:04.250406 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.250152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56866)
0810 18:16:04.250275 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.250335 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 18:16:04.251149 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.250892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56882)
0810 18:16:04.251036 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.251092 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260810 18:16:04.251304 27955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:38825: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
I20260810 18:16:04.282454 27955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } attempt: 1
I20260810 18:16:04.283344 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.282998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56894)
0810 18:16:04.283163 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.283243 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 18:16:04.284165 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.283931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56910)
0810 18:16:04.284041 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.284098 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 18:16:04.284304 27955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:38825: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
I20260810 18:16:04.289650 28075 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } attempt: 1
I20260810 18:16:04.290333 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.290061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56918)
0810 18:16:04.290195 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.290255 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 18:16:04.290974 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.290765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56920)
0810 18:16:04.290864 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.290913 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 18:16:04.291095 28075 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:38825: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
W20260810 18:16:04.348294 28083 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:16:04.348569 28083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:16:04.348592 28083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:16:04.348646 28083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:16:04.352708 28083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:16:04.352763 28083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:16:04.352782 28083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:16:04.352797 28083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:16:04.352824 28083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:16:04.357949 28083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:37137,127.23.65.125:42021,127.23.65.124:38825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab
--principal=kudu/127.23.65.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.124:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.65.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:16:04.358942 28083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:16:04.359236 28075 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } attempt: 2
I20260810 18:16:04.359445 27955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } attempt: 2
I20260810 18:16:04.360116 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.359816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56924)
0810 18:16:04.359952 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.360019 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 18:16:04.360173 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.359908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56930)
0810 18:16:04.360046 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.360098 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 18:16:04.360168 28083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:16:04.360898 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.360670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56946)
0810 18:16:04.360777 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.360833 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 18:16:04.361040 27955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:38825: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
I20260810 18:16:04.361119 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.360873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56960)
0810 18:16:04.360983 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.361036 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 18:16:04.361276 28075 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:38825: Network error: Client connection negotiation failed: client connection to 127.23.65.124:38825: connect: Connection refused (error 111)
W20260810 18:16:04.369021 28092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:04.369026 28093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:04.369024 28095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:04.369086 28083 server_base.cc:1034] running on GCE node
I20260810 18:16:04.369915 28083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:16:04.371052 28083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:16:04.372215 28083 hybrid_clock.cc:648] HybridClock initialized: now 1786385764372187 us; error 39 us; skew 500 ppm
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:16:04.374887 28083 init.cc:377] Logged in from keytab as kudu/127.23.65.124@KRBTEST.COM (short username kudu)
I20260810 18:16:04.375852 28083 webserver.cc:460] Webserver started at http://127.23.65.124:46795/ using document root <none> and password file <none>
I20260810 18:16:04.376391 28083 fs_manager.cc:359] Metadata directory not provided
I20260810 18:16:04.376461 28083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:16:04.376663 28083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:16:04.378712 28083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/data/instance:
uuid: "07c7e67137914661b686a0bff54db2be"
format_stamp: "Formatted at 2026-08-10 18:16:04 on dist-test-slave-jpj1"
server_key: "6f909978baae2a2e38fbf8cd168e332d"
server_key_iv: "436def6a7667a9b762ab03f930a973cd"
server_key_version: "encryptionkey@0"
I20260810 18:16:04.379324 28083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/wal/instance:
uuid: "07c7e67137914661b686a0bff54db2be"
format_stamp: "Formatted at 2026-08-10 18:16:04 on dist-test-slave-jpj1"
server_key: "6f909978baae2a2e38fbf8cd168e332d"
server_key_iv: "436def6a7667a9b762ab03f930a973cd"
server_key_version: "encryptionkey@0"
I20260810 18:16:04.383212 28083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:16:04.385519 28109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:16:04.386683 28083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:16:04.386816 28083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/wal
uuid: "07c7e67137914661b686a0bff54db2be"
format_stamp: "Formatted at 2026-08-10 18:16:04 on dist-test-slave-jpj1"
server_key: "6f909978baae2a2e38fbf8cd168e332d"
server_key_iv: "436def6a7667a9b762ab03f930a973cd"
server_key_version: "encryptionkey@0"
I20260810 18:16:04.386907 28083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:16:04.412161 28083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:16:04.414713 28083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:16:04.414891 28083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:16:04.421558 28083 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.124:38825
I20260810 18:16:04.421819 28196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.124:38825 every 8 connection(s)
I20260810 18:16:04.422505 28083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
I20260810 18:16:04.425231 28198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:16:04.428970 28198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:04.430207 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 28083
I20260810 18:16:04.430321 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385478759079-23813-0/minicluster-data/master-2/wal/instance
I20260810 18:16:04.430576 23813 external_mini_cluster.cc:1428] Setting key 45bab3529084000412d1d2e73ca41907
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
I20260810 18:16:04.441593 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.430813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.23.65.126:37137)
0810 18:16:04.430963 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.430966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.431980 (+ 1014us) server_negotiation.cc:407] Connection header received
0810 18:16:04.432651 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.432654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.432708 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.432775 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.433575 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.434224 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.435245 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.435473 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.435584 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.438009 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.438014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.438017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.438049 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.439845 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.440294 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.440298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.440300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.440380 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.440663 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.440666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.440669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.440839 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.440912 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.441304 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.441425 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:16:04.442536 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.431246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.23.65.124:38825)
0810 18:16:04.431673 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.431691 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.431733 (+ 42us) server_negotiation.cc:407] Connection header received
0810 18:16:04.432470 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.432488 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.432676 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.432840 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.433642 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.434465 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.435294 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.435692 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.435788 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.438378 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.438384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.438397 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.438421 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.439918 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.440302 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.440307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.440308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.440406 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.440658 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.440661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.440663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.441043 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.441255 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.441434 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.441579 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":299,"spinlock_wait_cycles":2304,"thread_start_us":129,"threads_started":1}
I20260810 18:16:04.442538 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.431248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:37137 (local address 127.0.0.1:57488)
0810 18:16:04.431862 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.431900 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:04.432476 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:04.432843 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:04.432848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:04.433188 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:04.433444 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.433457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.434357 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.434360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.435117 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.435127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.435632 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.435637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.435786 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.436315 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:04.436332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:04.437860 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:04.439942 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.439951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.439961 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.440188 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.440465 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.440468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.440470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.440579 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.440933 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:04.440939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:04.441058 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 18:16:04.441221 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 18:16:04.441531 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"spinlock_wait_cycles":7296,"thread_start_us":164,"threads_started":1}
I20260810 18:16:04.443832 28198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:16:04.452309 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.444419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:42021 (local address 127.0.0.1:36832)
0810 18:16:04.444554 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.444565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:04.444619 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:04.444853 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:04.444855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:04.445011 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:04.445123 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.445130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.446058 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.446061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.446807 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.446817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.447363 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.447367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.447463 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.447909 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:04.447924 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:04.449104 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:04.450840 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.450842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.450844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.451058 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.451356 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.451359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.451360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.451457 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.451881 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:04.451884 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:04.451983 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 18:16:04.452080 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 18:16:04.452191 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 18:16:04.452430 28075 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } attempt: 3
I20260810 18:16:04.452544 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.444528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.23.65.125:42021)
0810 18:16:04.444644 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.444649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.444663 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:16:04.444708 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.444711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.444760 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.444836 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.445247 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.445962 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.446916 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.447257 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.447341 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.449222 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.449225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.449227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.449249 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.450739 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.451170 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.451173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.451175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.451272 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.451572 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.451575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.451577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.451781 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.451876 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.452140 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.452277 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:16:04.453555 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.444827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.23.65.126:37137)
0810 18:16:04.444934 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.444938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.445025 (+ 87us) server_negotiation.cc:407] Connection header received
0810 18:16:04.445097 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.445100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.445138 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.445204 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.445869 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.446514 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.447471 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.447741 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.447833 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.450019 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.450023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.450026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.450052 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.451300 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.451724 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.451752 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.451753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.451836 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.452090 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.452093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.452094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.452305 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.452755 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.453324 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.453420 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:16:04.453527 28198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
I20260810 18:16:04.460193 27955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } attempt: 3
I20260810 18:16:04.460566 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.452786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56972)
0810 18:16:04.452922 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.452935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:04.452997 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:04.453218 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:04.453222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:04.453403 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:04.453496 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.453501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.454356 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.454359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.455026 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.455034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.455638 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.455642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.455734 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.456141 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:04.456154 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:04.457322 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:04.459094 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.459097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.459099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.459335 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.459619 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.459622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.459624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.459704 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.460111 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:04.460113 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:04.460192 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 18:16:04.460286 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 18:16:04.460408 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 18:16:04.460811 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.452855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.23.65.124:38825)
0810 18:16:04.452993 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.452997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.453010 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:16:04.453046 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.453048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.453085 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.453159 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.453599 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.454266 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.455149 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.455511 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.455633 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.457440 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.457443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.457446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.457470 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.458996 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.459431 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.459434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.459436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.459510 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.459800 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.459804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.459805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.460012 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.460117 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.460356 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.460672 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":768}
I20260810 18:16:04.463744 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.454601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.23.65.125:42021)
0810 18:16:04.454697 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.454701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.454714 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:16:04.454781 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.454783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.454819 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.454884 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.455656 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.456258 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.457297 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.457574 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.457649 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.460900 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.460904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.460906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.460933 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.462301 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.462660 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.462663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.462665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.462739 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.462977 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.462980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.462982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.463142 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.463347 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.463470 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.463604 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:16:04.464532 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04: Bootstrap starting.
Aug 10 18:16:04 dist-test-slave-jpj1 krb5kdc[27827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
I20260810 18:16:04.467190 28075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04: Neither blocks nor log segments found. Creating new log.
I20260810 18:16:04.468308 28075 log.cc:824] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04: Log is configured to *not* fsync() on all Append() calls
I20260810 18:16:04.469122 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.453929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56978)
0810 18:16:04.454020 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.454033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:04.454105 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:04.454825 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:04.454828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:04.454969 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:04.455026 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.455031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.456374 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.456377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.457159 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.457169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.457679 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.457682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.457750 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.458116 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:04.458130 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:04.459267 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:04.461542 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.461545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.461547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.461798 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.468126 (+ 6328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.468129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.468131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.468223 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.468666 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:04.468668 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:04.468743 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 18:16:04.468853 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 18:16:04.468975 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"spinlock_wait_cycles":25472}
I20260810 18:16:04.469339 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.460552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:38825 (local address 127.0.0.1:56988)
0810 18:16:04.460834 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:04.460848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:04.460920 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:04.462209 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:04.462212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:04.462388 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:04.462481 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.462488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.463434 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.463438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.464176 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:04.464187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.464739 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.464743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:04.464813 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.465151 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:04.465160 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:04.466195 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:04.467847 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.467850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.467852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.468105 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.468395 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:04.468399 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:04.468401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:04.468486 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:04.468877 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:04.468880 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:04.468972 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 18:16:04.469049 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 18:16:04.469165 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260810 18:16:04.469781 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:16:04.469744 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.454088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.23.65.124:38825)
0810 18:16:04.454648 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.454659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.454672 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:16:04.454702 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.454704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.454741 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.454816 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.455120 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.456259 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.457294 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.457587 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.457670 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.459408 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.459412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.459414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.459440 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.461440 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.461897 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.461901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.461903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.461993 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.468313 (+ 6320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.468316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.468318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.468555 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.468635 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.469344 (+ 709us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.469554 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"spinlock_wait_cycles":1408,"thread_start_us":128,"threads_started":1}
I20260810 18:16:04.469897 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.460583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.23.65.124:38825)
0810 18:16:04.462009 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.462013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.462025 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:16:04.462053 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.462055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.462090 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.462589 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.462667 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.463304 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.464332 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.464614 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.464708 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.466327 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.466331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.466333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.466357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.467740 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.468213 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.468216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.468218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.468296 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.468599 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.468602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.468603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.468765 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.469608 (+ 843us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.469679 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.469776 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1376,"thread_start_us":125,"threads_started":1}
I20260810 18:16:04.470558 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04: No bootstrap required, opened a new log
I20260810 18:16:04.473163 27955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588: Bootstrap starting.
I20260810 18:16:04.473572 28198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be: Bootstrap starting.
I20260810 18:16:04.474531 28075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } }
I20260810 18:16:04.474793 28075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fba6cd4de454857b12f03b21b88bc04, State: Initialized, Role: FOLLOWER
I20260810 18:16:04.475117 28075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } }
I20260810 18:16:04.475314 27955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588: Neither blocks nor log segments found. Creating new log.
I20260810 18:16:04.475657 28207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.475829 28207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:04.475980 28198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be: Neither blocks nor log segments found. Creating new log.
I20260810 18:16:04.476198 27955 log.cc:824] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588: Log is configured to *not* fsync() on all Append() calls
I20260810 18:16:04.476754 28198 log.cc:824] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be: Log is configured to *not* fsync() on all Append() calls
I20260810 18:16:04.476188 27911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57492
I20260810 18:16:04.478229 27955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588: No bootstrap required, opened a new log
I20260810 18:16:04.478610 28075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:16:04.478766 28198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be: No bootstrap required, opened a new log
I20260810 18:16:04.481660 27955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } }
I20260810 18:16:04.481884 27955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3f1b7eadd146398b8a3dfe85f1a588, State: Initialized, Role: FOLLOWER
I20260810 18:16:04.482728 28198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } }
I20260810 18:16:04.482949 28198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c7e67137914661b686a0bff54db2be, State: Initialized, Role: FOLLOWER
I20260810 18:16:04.483242 28198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } }
I20260810 18:16:04.483780 28229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.483982 28229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:04.484490 28198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:16:04.484530 28227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:16:04.484688 28227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:16:04.482280 27971 master_runner.cc:186] Error getting master registration for 127.23.65.126:37137: OK, instance_id { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" instance_seqno: 1786385763997778 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:16:04.485320 27971 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:16:04.482250 27955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } }
I20260810 18:16:04.485881 28241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.486018 28241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:04.486609 27955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:16:04.486907 27911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:16:04.487015 27911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:16:04.488304 27911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } }
I20260810 18:16:04.489233 27911 leader_election.cc:288] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [CANDIDATE]: Term 1 election: Requested vote from peers 6fba6cd4de454857b12f03b21b88bc04 (127.23.65.125:42021), 07c7e67137914661b686a0bff54db2be (127.23.65.124:38825)
I20260810 18:16:04.489964 28032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6fba6cd4de454857b12f03b21b88bc04"
I20260810 18:16:04.490180 28032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:16:04.491452 28032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3f1b7eadd146398b8a3dfe85f1a588 in term 1.
I20260810 18:16:04.487900 28083 master_runner.cc:186] Error getting master registration for 127.23.65.126:37137: OK, instance_id { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" instance_seqno: 1786385763997778 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:16:04.491770 28083 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:16:04.491777 28157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "07c7e67137914661b686a0bff54db2be"
I20260810 18:16:04.492070 28157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:16:04.493345 28157 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3f1b7eadd146398b8a3dfe85f1a588 in term 1.
W20260810 18:16:04.494587 27881 tablet.cc:2369] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:16:04.495146 28276 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 18:16:04.495141 27876 leader_election.cc:302] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c7e67137914661b686a0bff54db2be, 9b3f1b7eadd146398b8a3dfe85f1a588; no voters:
W20260810 18:16:04.495318 28276 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:16:04.495575 28241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:16:04.495783 28241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [term 1 LEADER]: Becoming Leader. State: Replica: 9b3f1b7eadd146398b8a3dfe85f1a588, State: Running, Role: LEADER
I20260810 18:16:04.496076 28241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } }
I20260810 18:16:04.497713 28280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b3f1b7eadd146398b8a3dfe85f1a588. Latest consensus state: current_term: 1 leader_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.497843 28280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [sys.catalog]: This master's current role is: LEADER
W20260810 18:16:04.498301 28250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:16:04.498374 28250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:16:04.501928 27846 master_runner.cc:263] No leader master found from master 9b3f1b7eadd146398b8a3dfe85f1a588
I20260810 18:16:04.501986 27846 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:16:04.509708 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.499938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.23.65.125:42021)
0810 18:16:04.500060 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.500065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.500521 (+ 456us) server_negotiation.cc:407] Connection header received
0810 18:16:04.500710 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.500713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.500762 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.500812 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.501834 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.502595 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.503894 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.504237 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.504317 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.506127 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.506131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.506133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.506159 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.507823 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.508313 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.508317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.508319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.508407 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.508715 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.508719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.508720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.508938 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.509042 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.509414 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.509529 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:16:04.510632 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.500618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.23.65.126:37137)
0810 18:16:04.500721 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.500725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.501588 (+ 863us) server_negotiation.cc:407] Connection header received
0810 18:16:04.501777 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.501780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.501825 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.501874 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.502846 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.503621 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.504777 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.505108 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.505181 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.507108 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.507112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.507115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.507142 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.508784 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.509272 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.509276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.509277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.509371 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.509703 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.509706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.509708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.509926 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.510022 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.510386 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.510498 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:16:04.516094 28282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:16:04.517073 28282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:16:04.518951 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:04.501049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.23.65.124:38825)
0810 18:16:04.501160 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.501164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.510363 (+ 9199us) server_negotiation.cc:407] Connection header received
0810 18:16:04.510532 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.510535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.510583 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.510631 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.511676 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.512418 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.513502 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:04.513813 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:04.513884 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:04.515484 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:04.515488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:04.515490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:04.515517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:04.517177 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.517686 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.517689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.517691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.517776 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:04.518050 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:04.518053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:04.518055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:04.518264 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:04.518357 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:04.518685 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 18:16:04.518798 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:16:04.523018 28157 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [term 1 FOLLOWER]: Refusing update from remote peer 9b3f1b7eadd146398b8a3dfe85f1a588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:16:04.523731 28280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:16:04.523880 28032 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [term 1 FOLLOWER]: Refusing update from remote peer 9b3f1b7eadd146398b8a3dfe85f1a588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:16:04.524415 28280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:16:04.528436 28229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b3f1b7eadd146398b8a3dfe85f1a588. Latest consensus state: current_term: 1 leader_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.528527 28229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:04.531708 28207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b3f1b7eadd146398b8a3dfe85f1a588. Latest consensus state: current_term: 1 leader_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.531840 28207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:04.534679 28280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.534801 28280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [sys.catalog]: This master's current role is: LEADER
I20260810 18:16:04.534915 28291 mvcc.cc:205] Tried to move back new op lower bound from 7317036091478654976 to 7317036091378606080. Current Snapshot: MvccSnapshot[applied={T|T < 7317036091478654976}]
I20260810 18:16:04.535408 28241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.535528 28241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588 [sys.catalog]: This master's current role is: LEADER
I20260810 18:16:04.536661 28292 mvcc.cc:205] Tried to move back new op lower bound from 7317036091478654976 to 7317036091378606080. Current Snapshot: MvccSnapshot[applied={T|T < 7317036091478654976}]
I20260810 18:16:04.536815 28229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.536912 28229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:04.537999 28207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3f1b7eadd146398b8a3dfe85f1a588" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 37137 } } peers { permanent_uuid: "6fba6cd4de454857b12f03b21b88bc04" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 42021 } } peers { permanent_uuid: "07c7e67137914661b686a0bff54db2be" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 38825 } } }
I20260810 18:16:04.538111 28207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:04.538479 28294 mvcc.cc:205] Tried to move back new op lower bound from 7317036091478654976 to 7317036091378606080. Current Snapshot: MvccSnapshot[applied={T|T < 7317036091478654976}]
I20260810 18:16:04.540189 28282 catalog_manager.cc:1347] Generated new cluster ID: c5f2a846639e4a8893625c77cb46af1c
I20260810 18:16:04.540256 28282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:16:04.570881 28282 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:16:04.572162 28282 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:16:04.583217 28282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b3f1b7eadd146398b8a3dfe85f1a588: Generated new TSK 0
I20260810 18:16:04.584193 28282 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:16:04.620223 28201 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.617613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.23.65.124:38825)
0810 18:16:04.617787 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.617792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.619053 (+ 1261us) server_negotiation.cc:407] Connection header received
0810 18:16:04.619268 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.619272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.619324 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.619378 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.620073 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57014: BlockingRecv error: recv got EOF from 127.0.0.1:57014 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 18:16:04.621577 27960 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.618422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.23.65.126:37137)
0810 18:16:04.618542 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.618547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.620723 (+ 2176us) server_negotiation.cc:407] Connection header received
0810 18:16:04.620888 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.620891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.620937 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.620983 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.621456 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57522: BlockingRecv error: recv got EOF from 127.0.0.1:57522 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 18:16:04.623243 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.622031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.23.65.125:42021)
0810 18:16:04.622148 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.622153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.622401 (+ 248us) server_negotiation.cc:407] Connection header received
0810 18:16:04.622551 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.622554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.622601 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.622647 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.623115 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36844: BlockingRecv error: recv got EOF from 127.0.0.1:36844 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 18:16:04.632869 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.627941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.23.65.125:42021)
0810 18:16:04.628051 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.628055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.631814 (+ 3759us) server_negotiation.cc:407] Connection header received
0810 18:16:04.632054 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.632057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.632141 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.632198 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.632758 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36856: BlockingRecv error: recv got EOF from 127.0.0.1:36856 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 18:16:04.637892 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.636423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.23.65.125:42021)
0810 18:16:04.636521 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.636526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.636998 (+ 472us) server_negotiation.cc:407] Connection header received
0810 18:16:04.637171 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.637174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.637222 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.637269 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.637789 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36858: BlockingRecv error: recv got EOF from 127.0.0.1:36858 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 18:16:04.643419 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.642044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.23.65.125:42021)
0810 18:16:04.642138 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.642142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.642577 (+ 435us) server_negotiation.cc:407] Connection header received
0810 18:16:04.642761 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.642764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.642809 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.642857 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.643322 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36860: BlockingRecv error: recv got EOF from 127.0.0.1:36860 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 18:16:04.651170 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.649619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.23.65.125:42021)
0810 18:16:04.649765 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.649770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.650245 (+ 475us) server_negotiation.cc:407] Connection header received
0810 18:16:04.650427 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.650431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.650480 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.650531 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.651066 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36862: BlockingRecv error: recv got EOF from 127.0.0.1:36862 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 18:16:04.663269 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.661614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.23.65.125:42021)
0810 18:16:04.661740 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.661745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.662263 (+ 518us) server_negotiation.cc:407] Connection header received
0810 18:16:04.662477 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.662480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.662535 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.662588 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.663162 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36866: BlockingRecv error: recv got EOF from 127.0.0.1:36866 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 18:16:04.683785 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.681980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.23.65.125:42021)
0810 18:16:04.682091 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.682095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.682660 (+ 565us) server_negotiation.cc:407] Connection header received
0810 18:16:04.682892 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.682895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.682946 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.682994 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.683658 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36874: BlockingRecv error: recv got EOF from 127.0.0.1:36874 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 18:16:04.719127 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.718092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.23.65.125:42021)
0810 18:16:04.718228 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.718232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.718330 (+ 98us) server_negotiation.cc:407] Connection header received
0810 18:16:04.718450 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.718454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.718510 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.718609 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.719002 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36876: BlockingRecv error: recv got EOF from 127.0.0.1:36876 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 18:16:04.786307 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.785159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.23.65.125:42021)
0810 18:16:04.785319 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.785325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.785453 (+ 128us) server_negotiation.cc:407] Connection header received
0810 18:16:04.785576 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.785578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.785631 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.785702 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.786179 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36888: BlockingRecv error: recv got EOF from 127.0.0.1:36888 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 18:16:04.917335 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:04.916233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.23.65.125:42021)
0810 18:16:04.916359 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:04.916363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:04.916496 (+ 133us) server_negotiation.cc:407] Connection header received
0810 18:16:04.916619 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:04.916621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:04.916669 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:04.916747 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:04.917207 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36892: BlockingRecv error: recv got EOF from 127.0.0.1:36892 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 18:16:05.176463 28081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:05.175335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.23.65.125:42021)
0810 18:16:05.175507 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:05.175510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:05.175637 (+ 127us) server_negotiation.cc:407] Connection header received
0810 18:16:05.175762 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:05.175764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:05.175822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:05.175894 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:05.176345 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36908: BlockingRecv error: recv got EOF from 127.0.0.1:36908 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 18:16:05.486191 28227 catalog_manager.cc:1259] Loaded cluster ID: c5f2a846639e4a8893625c77cb46af1c
I20260810 18:16:05.486274 28227 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04: loading cluster ID for follower catalog manager: success
I20260810 18:16:05.488107 28227 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04: acquiring CA information for follower catalog manager: success
I20260810 18:16:05.489305 28227 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fba6cd4de454857b12f03b21b88bc04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:16:05.499583 28250 catalog_manager.cc:1259] Loaded cluster ID: c5f2a846639e4a8893625c77cb46af1c
I20260810 18:16:05.499653 28250 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be: loading cluster ID for follower catalog manager: success
I20260810 18:16:05.501261 28250 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be: acquiring CA information for follower catalog manager: success
I20260810 18:16:05.502349 28250 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07c7e67137914661b686a0bff54db2be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:16:05.694226 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:05.690444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.23.65.125:42021)
0810 18:16:05.690711 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:05.690715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:05.690813 (+ 98us) server_negotiation.cc:407] Connection header received
0810 18:16:05.691005 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:05.691008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:05.691163 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:05.691280 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:05.691503 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.692083 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.692876 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.693102 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.693175 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:05.693423 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:05.693875 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:05.693961 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:05.693990 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 18:16:05.694041 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260810 18:16:05.700762 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:05.697316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.23.65.124:38825)
0810 18:16:05.697551 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:05.697554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:05.697567 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:16:05.697656 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:05.697658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:05.697776 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:05.697877 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:05.698036 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.698544 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.699303 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.699569 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.699694 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:05.699929 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:05.700369 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:05.700457 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:05.700560 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 18:16:05.700610 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260810 18:16:05.707608 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:05.703605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.23.65.125:42021)
0810 18:16:05.703725 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:05.703730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:05.704311 (+ 581us) server_negotiation.cc:407] Connection header received
0810 18:16:05.704414 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:05.704417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:05.704472 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:05.704552 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:05.704797 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.705552 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.706453 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.706696 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.706793 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:05.707125 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:05.707211 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:05.707281 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:05.707391 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 18:16:05.707447 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:16:05.708058 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:05.703942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.23.65.124:38825)
0810 18:16:05.704052 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:05.704056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:05.704070 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:16:05.704217 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:05.704221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:05.704287 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:05.704357 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:05.704649 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.705388 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.706684 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.706986 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.707089 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:05.707586 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:05.707662 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:05.707708 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:05.707872 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 18:16:05.707924 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:16:05.708379 28432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:05.703609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.23.65.126:37137)
0810 18:16:05.703885 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:05.703889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:05.704095 (+ 206us) server_negotiation.cc:407] Connection header received
0810 18:16:05.704518 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:05.704521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:05.704691 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:05.704808 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:05.705124 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.705668 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.706762 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:05.707004 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:05.707095 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:05.707523 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:05.707912 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:05.707970 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:05.708152 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 18:16:05.708222 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260810 18:16:05.715334 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 27846
I20260810 18:16:05.731577 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 27971
I20260810 18:16:05.744240 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 28083
2026-08-10T18:16:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3191 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:16:05 dist-test-slave-jpj1 krb5kdc[28439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:16:05 dist-test-slave-jpj1 krb5kdc[28439](info): set up 2 sockets
Aug 10 18:16:05 dist-test-slave-jpj1 krb5kdc[28439](info): commencing operation
krb5kdc: starting...
W20260810 18:16:06.897112 23813 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/test-user.keytab.
Aug 10 18:16:06 dist-test-slave-jpj1 krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.65.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
Entry for principal kudu/127.23.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab.
I20260810 18:16:07.018129 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43177
--webserver_interface=127.23.65.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:43177,127.23.65.125:45361,127.23.65.124:43659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:16:07.130317 28465 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:16:07.130654 28465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:16:07.130697 28465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:16:07.130762 28465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:16:07.134861 28465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:16:07.134927 28465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:16:07.134954 28465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:16:07.134974 28465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:16:07.134995 28465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:16:07.140348 28465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:43177,127.23.65.125:45361,127.23.65.124:43659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.126.keytab
--principal=kudu/127.23.65.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.126:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.65.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:16:07.141427 28465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:16:07.142467 28465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:16:07.150867 28473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:07.150908 28472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:07.150926 28465 server_base.cc:1034] running on GCE node
W20260810 18:16:07.150873 28477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:07.151646 28465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:16:07.152751 28465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:16:07.153915 28465 hybrid_clock.cc:648] HybridClock initialized: now 1786385767153899 us; error 29 us; skew 500 ppm
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:16:07.156724 28465 init.cc:377] Logged in from keytab as kudu/127.23.65.126@KRBTEST.COM (short username kudu)
I20260810 18:16:07.157614 28465 webserver.cc:460] Webserver started at http://127.23.65.126:36405/ using document root <none> and password file <none>
I20260810 18:16:07.158179 28465 fs_manager.cc:359] Metadata directory not provided
I20260810 18:16:07.158252 28465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:16:07.158460 28465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:16:07.160581 28465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/data/instance:
uuid: "ab310eff8ad14bbd9c148e3f4793772d"
format_stamp: "Formatted at 2026-08-10 18:16:07 on dist-test-slave-jpj1"
server_key: "eb27674e188f73e1ab06e39ef7df4cd0"
server_key_iv: "4a6beeed1b060ec99871b1e4f02c0e32"
server_key_version: "encryptionkey@0"
I20260810 18:16:07.161157 28465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/wal/instance:
uuid: "ab310eff8ad14bbd9c148e3f4793772d"
format_stamp: "Formatted at 2026-08-10 18:16:07 on dist-test-slave-jpj1"
server_key: "eb27674e188f73e1ab06e39ef7df4cd0"
server_key_iv: "4a6beeed1b060ec99871b1e4f02c0e32"
server_key_version: "encryptionkey@0"
I20260810 18:16:07.164798 28465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:16:07.167083 28493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:16:07.168310 28465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:16:07.168429 28465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/wal
uuid: "ab310eff8ad14bbd9c148e3f4793772d"
format_stamp: "Formatted at 2026-08-10 18:16:07 on dist-test-slave-jpj1"
server_key: "eb27674e188f73e1ab06e39ef7df4cd0"
server_key_iv: "4a6beeed1b060ec99871b1e4f02c0e32"
server_key_version: "encryptionkey@0"
I20260810 18:16:07.168524 28465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:16:07.190694 28465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:16:07.193606 28465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:16:07.193796 28465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:16:07.200665 28465 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.126:43177
I20260810 18:16:07.200932 28585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.126:43177 every 8 connection(s)
I20260810 18:16:07.201653 28465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/data/info.pb
I20260810 18:16:07.204416 28586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:16:07.204770 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 28465
I20260810 18:16:07.204888 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-0/wal/instance
I20260810 18:16:07.205147 23813 external_mini_cluster.cc:1428] Setting key c10d4d6432a559cb812cc9b4ddf566fa
I20260810 18:16:07.208660 28586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.65.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.125@KRBTEST.COM" created.
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:16:07.223775 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.210946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.23.65.126:43177)
0810 18:16:07.211385 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.211404 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.211604 (+ 200us) server_negotiation.cc:407] Connection header received
0810 18:16:07.212657 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.212664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.212924 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.213132 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.213879 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.215266 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.216273 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.216637 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.216755 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.219181 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.219187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.219198 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.219219 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.220748 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.221281 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.221285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.221287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.221394 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.221713 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.221715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.221716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.222087 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.222191 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.222671 (+ 480us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.222895 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":310,"spinlock_wait_cycles":40704,"thread_start_us":146,"threads_started":1}
I20260810 18:16:07.223776 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.210947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43177 (local address 127.0.0.1:57138)
0810 18:16:07.211425 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.211475 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:07.212343 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:07.213115 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:07.213119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:07.213452 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:07.213721 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.213734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.215384 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.215386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.216141 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.216151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.216776 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.216779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.216872 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.217429 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:07.217445 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:07.219036 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:07.220867 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.220876 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.220886 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.221165 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.221486 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.221489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.221491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.221635 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.222213 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:07.222221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:07.222398 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 18:16:07.222582 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 18:16:07.222969 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":61,"thread_start_us":117,"threads_started":1}
I20260810 18:16:07.226154 28586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:07.226912 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.226647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:45361 (local address 127.0.0.1:43670)
0810 18:16:07.226775 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.226849 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 18:16:07.227093 28496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
I20260810 18:16:07.228046 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.227830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:45361 (local address 127.0.0.1:43674)
0810 18:16:07.227928 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.227984 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 18:16:07.228184 28586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.125:45361: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab.
Entry for principal kudu/127.23.65.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab.
I20260810 18:16:07.234007 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.125:45361
--webserver_interface=127.23.65.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab
--principal=kudu/127.23.65.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:43177,127.23.65.125:45361,127.23.65.124:43659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:16:07.277467 28586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } attempt: 1
I20260810 18:16:07.278376 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.278069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:45361 (local address 127.0.0.1:43680)
0810 18:16:07.278226 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.278291 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 18:16:07.279148 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.278934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:45361 (local address 127.0.0.1:43690)
0810 18:16:07.279025 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.279074 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 18:16:07.279269 28586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.125:45361: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
W20260810 18:16:07.346844 28601 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:16:07.347121 28601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:16:07.347143 28601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:16:07.347189 28601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 18:16:07.347414 28586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } attempt: 2
I20260810 18:16:07.348241 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.347942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:45361 (local address 127.0.0.1:43698)
0810 18:16:07.348086 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.348146 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 18:16:07.348933 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.348737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:45361 (local address 127.0.0.1:43708)
0810 18:16:07.348826 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.348873 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 18:16:07.349051 28586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.125:45361: Network error: Client connection negotiation failed: client connection to 127.23.65.125:45361: connect: Connection refused (error 111)
W20260810 18:16:07.351511 28601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:16:07.351591 28601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:16:07.351612 28601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:16:07.351634 28601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:16:07.351652 28601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:16:07.356874 28601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:43177,127.23.65.125:45361,127.23.65.124:43659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.125.keytab
--principal=kudu/127.23.65.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.125:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.65.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:16:07.357839 28601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:16:07.358762 28601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:16:07.366822 28615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:07.366839 28613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:07.366878 28611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:07.367283 28601 server_base.cc:1034] running on GCE node
I20260810 18:16:07.367852 28601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:16:07.368849 28601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:16:07.370007 28601 hybrid_clock.cc:648] HybridClock initialized: now 1786385767369990 us; error 28 us; skew 500 ppm
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:16:07.372668 28601 init.cc:377] Logged in from keytab as kudu/127.23.65.125@KRBTEST.COM (short username kudu)
I20260810 18:16:07.373508 28601 webserver.cc:460] Webserver started at http://127.23.65.125:46065/ using document root <none> and password file <none>
I20260810 18:16:07.374019 28601 fs_manager.cc:359] Metadata directory not provided
I20260810 18:16:07.374078 28601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:16:07.374260 28601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:16:07.376283 28601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/data/instance:
uuid: "7995c4b4985a427bb5dc6563d0a7f562"
format_stamp: "Formatted at 2026-08-10 18:16:07 on dist-test-slave-jpj1"
server_key: "06c1ca4d465392ec740d3719a6c6d7ba"
server_key_iv: "1e279431ed898608a2bbe734cc61ca81"
server_key_version: "encryptionkey@0"
I20260810 18:16:07.376811 28601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/wal/instance:
uuid: "7995c4b4985a427bb5dc6563d0a7f562"
format_stamp: "Formatted at 2026-08-10 18:16:07 on dist-test-slave-jpj1"
server_key: "06c1ca4d465392ec740d3719a6c6d7ba"
server_key_iv: "1e279431ed898608a2bbe734cc61ca81"
server_key_version: "encryptionkey@0"
I20260810 18:16:07.380388 28601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:16:07.382710 28625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:16:07.383978 28601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:16:07.384099 28601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/wal
uuid: "7995c4b4985a427bb5dc6563d0a7f562"
format_stamp: "Formatted at 2026-08-10 18:16:07 on dist-test-slave-jpj1"
server_key: "06c1ca4d465392ec740d3719a6c6d7ba"
server_key_iv: "1e279431ed898608a2bbe734cc61ca81"
server_key_version: "encryptionkey@0"
I20260810 18:16:07.384202 28601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:16:07.418256 28601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:16:07.421591 28601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:16:07.421787 28601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:16:07.429065 28601 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.125:45361
I20260810 18:16:07.429369 28702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.125:45361 every 8 connection(s)
I20260810 18:16:07.430073 28601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/data/info.pb
I20260810 18:16:07.431070 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 28601
I20260810 18:16:07.431197 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-1/wal/instance
I20260810 18:16:07.431504 23813 external_mini_cluster.cc:1428] Setting key 2cebe0676c79b8c65e271d338cecfd90
I20260810 18:16:07.433550 28703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:16:07.437523 28703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.65.124@KRBTEST.COM; defaulting to no policy
I20260810 18:16:07.440191 28586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.65.124@KRBTEST.COM" created.
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
I20260810 18:16:07.452037 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.440632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:45361 (local address 127.0.0.1:43720)
0810 18:16:07.440865 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.440878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:07.440959 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:07.442240 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:07.442243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:07.442405 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:07.442510 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.442516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.444349 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.444352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.445168 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.445180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.445998 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.446002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.446096 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.446571 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:07.446586 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:07.448052 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:07.450213 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.450216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.450219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.450431 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.450750 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.450753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.450755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.450841 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.451466 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:07.451469 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:07.451601 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 18:16:07.451711 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 18:16:07.451856 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260810 18:16:07.453334 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.439572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.23.65.126:43177)
0810 18:16:07.439705 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.439709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.440854 (+ 1145us) server_negotiation.cc:407] Connection header received
0810 18:16:07.442010 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.442013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.442061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.442135 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.443260 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.444001 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.445403 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.445708 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.445817 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.449076 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.449080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.449083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.449112 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.451519 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.452061 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.452065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.452067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.452152 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.452443 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.452447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.452449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.452641 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.452698 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.453066 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.453172 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:16:07.453387 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.440089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43177 (local address 127.0.0.1:57152)
0810 18:16:07.440674 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.440734 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:07.441784 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:07.442206 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:07.442212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:07.442656 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:07.443001 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.443016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.444131 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.444134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.445301 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.445312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.445872 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.445877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.446066 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.446743 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:07.446760 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:07.448925 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:07.451648 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.451655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.451662 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.451948 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.452261 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.452263 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.452265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.452346 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.452776 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:07.452781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:07.452878 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 18:16:07.452991 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 18:16:07.453171 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":47,"spinlock_wait_cycles":1280,"thread_start_us":156,"threads_started":1}
I20260810 18:16:07.453385 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.441043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.23.65.125:45361)
0810 18:16:07.441362 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.441380 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.441425 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:16:07.441787 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.441811 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.442068 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.442236 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.442760 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.444219 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.445288 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.445842 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.446060 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.448227 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.448235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.448250 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.448277 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.450110 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.450536 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.450542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.450549 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.450660 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.450923 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.450927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.450928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.451363 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.451484 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.451909 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.452236 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":178,"spinlock_wait_cycles":3328,"thread_start_us":82,"threads_started":1}
I20260810 18:16:07.455150 28703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:07.456178 28586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:07.456899 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.456634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45380)
0810 18:16:07.456744 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.456816 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 18:16:07.457683 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.457425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45392)
0810 18:16:07.457560 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.457608 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 18:16:07.457860 28586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:43659: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.65.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab.
Entry for principal kudu/127.23.65.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab.
I20260810 18:16:07.463253 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.455731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:45361 (local address 127.0.0.1:43730)
0810 18:16:07.455855 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.455869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:07.455943 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:07.456168 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:07.456171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:07.456334 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:07.456472 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.456480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.457271 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.457275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.457816 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.457826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.458278 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.458281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.458403 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.458876 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:07.458890 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:07.460340 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:07.461871 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.461875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.461877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.462060 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.462313 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.462316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.462318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.462393 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.462734 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:07.462737 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:07.462825 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 18:16:07.462908 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 18:16:07.463055 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":7}
I20260810 18:16:07.463253 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.455822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.23.65.125:45361)
0810 18:16:07.455937 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.455941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.455955 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:16:07.456018 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.456021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.456065 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.456144 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.456575 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.457170 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.457915 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.458181 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.458269 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.460480 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.460484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.460486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.460510 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.461743 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.462153 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.462156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.462158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.462233 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.462467 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.462470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.462471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.462652 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.462734 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.462976 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.463112 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":42}
I20260810 18:16:07.463356 23813 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
/tmp/dist-test-taskmz0U2F/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.124:43659
--webserver_interface=127.23.65.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab
--principal=kudu/127.23.65.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.65.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.65.126:43177,127.23.65.125:45361,127.23.65.124:43659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:16:07.464042 28703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:07.464756 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.464480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45406)
0810 18:16:07.464590 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.464688 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 18:16:07.465051 28626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
I20260810 18:16:07.465999 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.465781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45418)
0810 18:16:07.465894 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.465942 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:16:07.466142 28703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:43659: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
I20260810 18:16:07.489015 28586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } attempt: 1
I20260810 18:16:07.489918 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.489595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45424)
0810 18:16:07.489761 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.489831 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 18:16:07.490708 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.490466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45426)
0810 18:16:07.490587 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.490639 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 18:16:07.490828 28586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:43659: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
I20260810 18:16:07.515451 28703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } attempt: 1
I20260810 18:16:07.516474 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.516138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45434)
0810 18:16:07.516315 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.516385 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 18:16:07.517284 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.517058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45450)
0810 18:16:07.517176 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.517223 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 18:16:07.517416 28703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:43659: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
I20260810 18:16:07.565981 28586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } attempt: 2
I20260810 18:16:07.566821 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.566523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45454)
0810 18:16:07.566672 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.566738 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 18:16:07.567498 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.567294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45460)
0810 18:16:07.567394 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.567440 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 18:16:07.567627 28586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:43659: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
W20260810 18:16:07.580715 28716 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:16:07.581025 28716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:16:07.581094 28716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:16:07.581161 28716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:16:07.585129 28716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:16:07.585178 28716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:16:07.585204 28716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:16:07.585227 28716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:16:07.585254 28716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:16:07.585554 28703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } attempt: 2
I20260810 18:16:07.586241 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.585982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45472)
0810 18:16:07.586112 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.586170 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 18:16:07.586935 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.586737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45480)
0810 18:16:07.586835 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.586881 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 18:16:07.587042 28703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.65.124:43659: Network error: Client connection negotiation failed: client connection to 127.23.65.124:43659: connect: Connection refused (error 111)
I20260810 18:16:07.590622 28716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.65.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.65.126:43177,127.23.65.125:45361,127.23.65.124:43659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/krb5kdc/kudu_127.23.65.124.keytab
--principal=kudu/127.23.65.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.65.124:43659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.65.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:16:07.591676 28716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:16:07.592666 28716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:16:07.600636 28733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:07.600654 28736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:16:07.600647 28734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:16:07.600806 28716 server_base.cc:1034] running on GCE node
I20260810 18:16:07.601444 28716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:16:07.602489 28716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:16:07.603592 28716 hybrid_clock.cc:648] HybridClock initialized: now 1786385767603571 us; error 33 us; skew 500 ppm
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:16:07.606271 28716 init.cc:377] Logged in from keytab as kudu/127.23.65.124@KRBTEST.COM (short username kudu)
I20260810 18:16:07.607079 28716 webserver.cc:460] Webserver started at http://127.23.65.124:37519/ using document root <none> and password file <none>
I20260810 18:16:07.607632 28716 fs_manager.cc:359] Metadata directory not provided
I20260810 18:16:07.607705 28716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:16:07.607898 28716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:16:07.609861 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/data/instance:
uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f"
format_stamp: "Formatted at 2026-08-10 18:16:07 on dist-test-slave-jpj1"
server_key: "1132b6c17a403b062f8bf62ae255d38f"
server_key_iv: "b36ec8b674964b28ebc656f002339904"
server_key_version: "encryptionkey@0"
I20260810 18:16:07.610399 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/wal/instance:
uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f"
format_stamp: "Formatted at 2026-08-10 18:16:07 on dist-test-slave-jpj1"
server_key: "1132b6c17a403b062f8bf62ae255d38f"
server_key_iv: "b36ec8b674964b28ebc656f002339904"
server_key_version: "encryptionkey@0"
I20260810 18:16:07.613885 28716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:16:07.616139 28749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:16:07.617297 28716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:16:07.617409 28716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/data,/tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/wal
uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f"
format_stamp: "Formatted at 2026-08-10 18:16:07 on dist-test-slave-jpj1"
server_key: "1132b6c17a403b062f8bf62ae255d38f"
server_key_iv: "b36ec8b674964b28ebc656f002339904"
server_key_version: "encryptionkey@0"
I20260810 18:16:07.617499 28716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:16:07.640707 28716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:16:07.644105 28716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:16:07.644318 28716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:16:07.651058 28716 rpc_server.cc:307] RPC server started. Bound to: 127.23.65.124:43659
I20260810 18:16:07.651335 28837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.65.124:43659 every 8 connection(s)
I20260810 18:16:07.652024 28716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/data/info.pb
I20260810 18:16:07.654727 28839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:16:07.658417 28839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:07.661167 23813 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu as pid 28716
I20260810 18:16:07.661312 23813 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmz0U2F/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385478759079-23813-0/minicluster-data/master-2/wal/instance
I20260810 18:16:07.661559 23813 external_mini_cluster.cc:1428] Setting key 3b189ceb506a112c05a1dc00c87ff9a5
I20260810 18:16:07.666778 28586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } attempt: 3
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.126@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
I20260810 18:16:07.673517 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.660171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.23.65.126:43177)
0810 18:16:07.660288 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.660291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.661172 (+ 881us) server_negotiation.cc:407] Connection header received
0810 18:16:07.662124 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.662127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.662190 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.662266 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.663865 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.664632 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.666031 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.666401 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.666507 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.669309 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.669313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.669315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.669342 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.670984 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.671628 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.671631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.671633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.671722 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.672113 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.672117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.672119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.672322 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.672423 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.673020 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.673359 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:16:07.674291 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.662195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.23.65.124:43659)
0810 18:16:07.662561 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.662569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.662632 (+ 63us) server_negotiation.cc:407] Connection header received
0810 18:16:07.662751 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.662755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.662902 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.663027 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.663947 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.665276 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.666350 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.666780 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.666875 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.669907 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.669914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.669926 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.669952 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.671861 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.672305 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.672311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.672313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.672415 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.672698 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.672701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.672703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.673013 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.673916 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.674006 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.674115 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":287,"thread_start_us":143,"threads_started":1}
I20260810 18:16:07.674299 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.660512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.126:43177 (local address 127.0.0.1:57160)
0810 18:16:07.661000 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.661044 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:07.661914 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:07.662451 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:07.662460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:07.662915 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:07.663666 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.663688 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.664739 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.664742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.665865 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.665876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.666544 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.666548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.666756 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.667558 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:07.667580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:07.669184 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:07.671096 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.671106 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.671125 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.671487 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.671848 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.671851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.671853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.672016 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.672436 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:07.672443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:07.672763 (+ 320us) client_negotiation.cc:769] Sending connection context
0810 18:16:07.672963 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 18:16:07.673265 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":61,"thread_start_us":122,"threads_started":1}
I20260810 18:16:07.675522 28839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:07.676460 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.667206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45496)
0810 18:16:07.667325 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.667340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:07.667416 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:07.667891 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:07.667894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:07.668062 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:07.668177 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.668184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.668968 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.668972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.669787 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.669797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.670284 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.670288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.670378 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.670695 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:07.670710 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:07.672093 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:07.674586 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.674589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.674591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.674825 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.675454 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.675457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.675459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.675583 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.676043 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:07.676046 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:07.676146 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 18:16:07.676228 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:16:07.676345 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 18:16:07.677225 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.667248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.23.65.124:43659)
0810 18:16:07.667651 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.667662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.667675 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:16:07.667708 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.667710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.667748 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.668289 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.668372 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.668845 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.669909 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.670185 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.670281 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.672253 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.672257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.672260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.672284 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.674461 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.674941 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.674943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.674945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.675028 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.675702 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.675705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.675706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.675913 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.676865 (+ 952us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.676935 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.677052 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":82,"threads_started":1}
I20260810 18:16:07.678185 28703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } attempt: 3
I20260810 18:16:07.680935 28586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d: Bootstrap starting.
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.126@KRBTEST.COM
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.125@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
I20260810 18:16:07.684623 28586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d: Neither blocks nor log segments found. Creating new log.
I20260810 18:16:07.685580 28586 log.cc:824] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:16:07.685989 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.676831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.23.65.126:43177)
0810 18:16:07.676940 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.676944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.677015 (+ 71us) server_negotiation.cc:407] Connection header received
0810 18:16:07.677135 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.677138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.677177 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.677253 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.677917 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.678506 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.679502 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.679798 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.679872 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.682564 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.682567 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.682570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.682592 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.684128 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.684608 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.684625 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.684627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.684726 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.685040 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.685043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.685044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.685233 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.685380 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.685756 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.685859 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:16:07.686010 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.676077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.23.65.125:45361)
0810 18:16:07.676183 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.676188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.676726 (+ 538us) server_negotiation.cc:407] Connection header received
0810 18:16:07.676828 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.676831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.676887 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.676963 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.677481 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.678170 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.679127 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.679451 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.679522 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.681824 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.681828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.681830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.681854 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.683370 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.684607 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.684611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.684612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.684711 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.685013 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.685017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.685018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.685249 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.685335 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.685739 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.685853 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":30}
I20260810 18:16:07.686434 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.675972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.125:45361 (local address 127.0.0.1:43732)
0810 18:16:07.676655 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.676668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:07.676749 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:07.677077 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:07.677080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:07.677245 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:07.677347 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.677354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.678298 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.678301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.678986 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.678997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.679887 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.679891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.679983 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.680364 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:07.680378 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:07.681680 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:07.684236 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.684240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.684242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.684491 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.684811 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.684814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.684815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.684899 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.685412 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:07.685414 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:07.685510 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 18:16:07.686141 (+ 631us) client_negotiation.cc:240] Negotiation successful
0810 18:16:07.686298 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608}
I20260810 18:16:07.687870 28586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d: No bootstrap required, opened a new log
I20260810 18:16:07.688216 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.678532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45512)
0810 18:16:07.678777 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.678790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:07.678858 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:07.679317 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:07.679320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:07.679496 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:07.679727 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.679733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.680606 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.680610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.681442 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.681454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.682121 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.682125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.682208 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.682824 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:07.682838 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:07.684620 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:07.686493 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.686496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.686498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.686732 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.687194 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.687198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.687200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.687285 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.687712 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:07.687715 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:07.687826 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 18:16:07.687908 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:16:07.688045 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":81}
I20260810 18:16:07.688609 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.679030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.23.65.124:43659)
0810 18:16:07.679121 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.679124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.679135 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:16:07.679165 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.679167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.679200 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.679603 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.679830 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.680473 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.681588 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.681972 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.682411 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.684751 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.684755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.684757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.684779 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.686374 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.686963 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.686966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.686968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.687057 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.687390 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.687393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.687395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.687584 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.688366 (+ 782us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.688427 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.688509 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:16:07.691005 28703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562: Bootstrap starting.
I20260810 18:16:07.692080 28839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:16:07.692576 28586 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } }
I20260810 18:16:07.692987 28586 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab310eff8ad14bbd9c148e3f4793772d, State: Initialized, Role: FOLLOWER
I20260810 18:16:07.693368 28586 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } }
I20260810 18:16:07.693459 28703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562: Neither blocks nor log segments found. Creating new log.
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.65.125@KRBTEST.COM
I20260810 18:16:07.693915 28850 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.694137 28850 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:07.694270 28703 log.cc:824] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562: Log is configured to *not* fsync() on all Append() calls
I20260810 18:16:07.694800 28586 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:16:07.695319 28500 tablet.cc:2369] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:16:07.696329 28703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562: No bootstrap required, opened a new log
I20260810 18:16:07.697510 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.687815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.23.65.125:45361)
0810 18:16:07.687915 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.687918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.687956 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:16:07.688082 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.688085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.688126 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.688203 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.689054 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.689759 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.690803 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.691092 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.691196 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.694043 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.694046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.694049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.694073 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.695600 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.696093 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.696096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.696098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.696187 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.696513 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.696517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.696519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.696737 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.696905 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.697266 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.697373 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:16:07.700858 28703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } }
I20260810 18:16:07.701076 28703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7995c4b4985a427bb5dc6563d0a7f562, State: Initialized, Role: FOLLOWER
I20260810 18:16:07.701156 23813 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 10 18:16:07 dist-test-slave-jpj1 krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385767, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.65.124@KRBTEST.COM for kudu/127.23.65.124@KRBTEST.COM
I20260810 18:16:07.701404 28703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } }
I20260810 18:16:07.702438 28536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ab310eff8ad14bbd9c148e3f4793772d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168
I20260810 18:16:07.702665 28536 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:16:07.702720 28703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:16:07.702777 28536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:16:07.700202 28465 master_runner.cc:186] Error getting master registration for 127.23.65.125:45361: OK, instance_id { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" instance_seqno: 1786385767427021 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:16:07.703958 28465 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:16:07.704106 28536 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } }
I20260810 18:16:07.704850 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.692524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.65.124:43659 (local address 127.0.0.1:45526)
0810 18:16:07.692725 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 18:16:07.692737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:16:07.692803 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:16:07.697434 (+ 4631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:16:07.697437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:16:07.697586 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:16:07.697685 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.697692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.698580 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.698583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.699265 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:16:07.699276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.699826 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.699830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:16:07.699901 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.700303 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:16:07.700318 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:16:07.701687 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:16:07.703457 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.703460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.703462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.703705 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.703975 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:16:07.703978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:16:07.703980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:16:07.704061 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:16:07.704448 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:16:07.704451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:16:07.704522 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 18:16:07.704593 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 18:16:07.704709 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260810 18:16:07.705070 28536 leader_election.cc:288] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [CANDIDATE]: Term 1 election: Requested vote from peers 7995c4b4985a427bb5dc6563d0a7f562 (127.23.65.125:45361), b4b9c86eb1c04a7e90ee19df2bd7e51f (127.23.65.124:43659)
I20260810 18:16:07.705324 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.692737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.23.65.124:43659)
0810 18:16:07.697249 (+ 4512us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.697253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.697264 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:16:07.697293 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.697295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.697332 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.697776 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.697836 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.698480 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.699387 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.699715 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.700062 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.701809 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.701811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.701813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.701833 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.703354 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.703810 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.703812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.703814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.703889 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.704167 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.704170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.704172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.704355 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.705016 (+ 661us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.705089 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.705201 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4441}
I20260810 18:16:07.706564 28665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab310eff8ad14bbd9c148e3f4793772d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7995c4b4985a427bb5dc6563d0a7f562"
I20260810 18:16:07.706795 28665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:16:07.708091 28665 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab310eff8ad14bbd9c148e3f4793772d in term 1.
I20260810 18:16:07.708107 28839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f: Bootstrap starting.
I20260810 18:16:07.708490 28496 leader_election.cc:302] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7995c4b4985a427bb5dc6563d0a7f562, ab310eff8ad14bbd9c148e3f4793772d; no voters:
I20260810 18:16:07.708730 28850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:16:07.708940 28850 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [term 1 LEADER]: Becoming Leader. State: Replica: ab310eff8ad14bbd9c148e3f4793772d, State: Running, Role: LEADER
I20260810 18:16:07.709296 28850 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } }
I20260810 18:16:07.710650 28839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f: Neither blocks nor log segments found. Creating new log.
W20260810 18:16:07.711120 28867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:16:07.711225 28867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:16:07.711453 28839 log.cc:824] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:16:07.711768 28895 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab310eff8ad14bbd9c148e3f4793772d. Latest consensus state: current_term: 1 leader_uuid: "ab310eff8ad14bbd9c148e3f4793772d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.711964 28895 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [sys.catalog]: This master's current role is: LEADER
I20260810 18:16:07.712311 28868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.712491 28868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:07.710429 28601 master_runner.cc:186] Error getting master registration for 127.23.65.124:43659: OK, instance_id { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" instance_seqno: 1786385767649185 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:16:07.714124 28601 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:16:07.714785 28899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260810 18:16:07.715754 28888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:16:07.715869 28888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:16:07.715934 28839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f: No bootstrap required, opened a new log
I20260810 18:16:07.716372 28899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:16:07.719844 28839 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } }
I20260810 18:16:07.720082 28839 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b9c86eb1c04a7e90ee19df2bd7e51f, State: Initialized, Role: FOLLOWER
I20260810 18:16:07.720474 28839 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } }
I20260810 18:16:07.721160 28902 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.721433 28902 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:07.721932 28839 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:16:07.721966 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab310eff8ad14bbd9c148e3f4793772d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f"
I20260810 18:16:07.722194 28796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:16:07.724182 28796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab310eff8ad14bbd9c148e3f4793772d in term 1.
W20260810 18:16:07.730683 28927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:16:07.730822 28927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:16:07.740618 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.720499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.23.65.126:43177)
0810 18:16:07.720603 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.720607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.726253 (+ 5646us) server_negotiation.cc:407] Connection header received
0810 18:16:07.731439 (+ 5186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.731442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.731496 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.731569 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.732442 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.733115 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.734952 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.735296 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.735374 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.737068 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.737072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.737075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.737100 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.738758 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.739254 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.739258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.739260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.739346 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.739673 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.739677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.739679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.739886 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.739988 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.740341 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.740455 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:16:07.749176 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.721911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.23.65.125:45361)
0810 18:16:07.722017 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.722021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.730866 (+ 8845us) server_negotiation.cc:407] Connection header received
0810 18:16:07.731037 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.731041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.731087 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.731135 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.732093 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.732826 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.733885 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.734202 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.734268 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.743667 (+ 9399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.743671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.743674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.743705 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.746967 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.747606 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.747610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.747611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.747698 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.747979 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.747981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.747983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.748215 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.748324 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.748893 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.749005 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:16:07.750466 28796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [term 1 FOLLOWER]: Refusing update from remote peer ab310eff8ad14bbd9c148e3f4793772d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:16:07.751066 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:07.720957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.23.65.124:43659)
0810 18:16:07.721060 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.721064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.723646 (+ 2582us) server_negotiation.cc:407] Connection header received
0810 18:16:07.724608 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.724611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.724659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.724711 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.726401 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.727167 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.734117 (+ 6950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:07.734436 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:07.734502 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:07.744061 (+ 9559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:16:07.744065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:16:07.744068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:16:07.744096 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:16:07.745995 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.746466 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.746470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.746471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.746554 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:16:07.746832 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:16:07.746835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:16:07.746837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:16:07.747041 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:16:07.747192 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:07.750811 (+ 3619us) server_negotiation.cc:299] Negotiation successful
0810 18:16:07.750922 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:16:07.751099 28895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:16:07.755947 28902 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab310eff8ad14bbd9c148e3f4793772d. Latest consensus state: current_term: 1 leader_uuid: "ab310eff8ad14bbd9c148e3f4793772d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.756071 28902 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:07.757056 28665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [term 1 FOLLOWER]: Refusing update from remote peer ab310eff8ad14bbd9c148e3f4793772d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:16:07.757753 28895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:16:07.764820 28868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab310eff8ad14bbd9c148e3f4793772d. Latest consensus state: current_term: 1 leader_uuid: "ab310eff8ad14bbd9c148e3f4793772d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.764966 28868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:07.772514 28895 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab310eff8ad14bbd9c148e3f4793772d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.772677 28895 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [sys.catalog]: This master's current role is: LEADER
I20260810 18:16:07.775633 28868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab310eff8ad14bbd9c148e3f4793772d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.775777 28868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:07.776371 28901 mvcc.cc:205] Tried to move back new op lower bound from 7317036104698064896 to 7317036104540585984. Current Snapshot: MvccSnapshot[applied={T|T < 7317036104698064896}]
I20260810 18:16:07.778180 28933 mvcc.cc:205] Tried to move back new op lower bound from 7317036104698064896 to 7317036104540585984. Current Snapshot: MvccSnapshot[applied={T|T < 7317036104698064896}]
I20260810 18:16:07.778898 28850 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab310eff8ad14bbd9c148e3f4793772d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.779034 28850 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d [sys.catalog]: This master's current role is: LEADER
I20260810 18:16:07.780158 28902 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab310eff8ad14bbd9c148e3f4793772d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab310eff8ad14bbd9c148e3f4793772d" member_type: VOTER last_known_addr { host: "127.23.65.126" port: 43177 } } peers { permanent_uuid: "7995c4b4985a427bb5dc6563d0a7f562" member_type: VOTER last_known_addr { host: "127.23.65.125" port: 45361 } } peers { permanent_uuid: "b4b9c86eb1c04a7e90ee19df2bd7e51f" member_type: VOTER last_known_addr { host: "127.23.65.124" port: 43659 } } }
I20260810 18:16:07.780282 28902 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:16:07.781073 28929 mvcc.cc:205] Tried to move back new op lower bound from 7317036104698064896 to 7317036104540585984. Current Snapshot: MvccSnapshot[applied={T|T < 7317036104698064896}]
I20260810 18:16:07.786931 28899 catalog_manager.cc:1347] Generated new cluster ID: 5dfa4306107b46368266bf019a7d8eb4
I20260810 18:16:07.786998 28899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:16:07.812492 28899 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:16:07.816533 28899 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:16:07.840517 28899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab310eff8ad14bbd9c148e3f4793772d: Generated new TSK 0
I20260810 18:16:07.841322 28899 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:16:07.876832 28597 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.873562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.23.65.126:43177)
0810 18:16:07.873698 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.873703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.875669 (+ 1966us) server_negotiation.cc:407] Connection header received
0810 18:16:07.875894 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.875898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.875955 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.876011 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.876684 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57182: BlockingRecv error: recv got EOF from 127.0.0.1:57182 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 18:16:07.878355 28847 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.874827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.23.65.124:43659)
0810 18:16:07.874965 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.874969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.877377 (+ 2408us) server_negotiation.cc:407] Connection header received
0810 18:16:07.877567 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.877570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.877623 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.877679 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.878226 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45548: BlockingRecv error: recv got EOF from 127.0.0.1:45548 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 18:16:07.879805 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.874368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.23.65.125:45361)
0810 18:16:07.874489 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.874493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.878835 (+ 4342us) server_negotiation.cc:407] Connection header received
0810 18:16:07.879002 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.879005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.879056 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.879106 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.879682 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43754: BlockingRecv error: recv got EOF from 127.0.0.1:43754 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 18:16:07.884187 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.882683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.23.65.125:45361)
0810 18:16:07.882783 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.882787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.883273 (+ 486us) server_negotiation.cc:407] Connection header received
0810 18:16:07.883460 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.883463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.883577 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.883623 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.884086 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43764: BlockingRecv error: recv got EOF from 127.0.0.1:43764 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 18:16:07.889065 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.887573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.23.65.125:45361)
0810 18:16:07.887667 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.887671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.888141 (+ 470us) server_negotiation.cc:407] Connection header received
0810 18:16:07.888309 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.888312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.888356 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.888401 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.888971 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43770: BlockingRecv error: recv got EOF from 127.0.0.1:43770 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 18:16:07.894433 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.893012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.23.65.125:45361)
0810 18:16:07.893113 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.893116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.893608 (+ 492us) server_negotiation.cc:407] Connection header received
0810 18:16:07.893786 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.893789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.893835 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.893878 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.894342 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43786: BlockingRecv error: recv got EOF from 127.0.0.1:43786 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 18:16:07.902004 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.900558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.23.65.125:45361)
0810 18:16:07.900682 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.900686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.901152 (+ 466us) server_negotiation.cc:407] Connection header received
0810 18:16:07.901327 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.901330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.901376 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.901420 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.901902 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43800: BlockingRecv error: recv got EOF from 127.0.0.1:43800 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 18:16:07.913163 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.912079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.23.65.125:45361)
0810 18:16:07.912187 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.912191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.912403 (+ 212us) server_negotiation.cc:407] Connection header received
0810 18:16:07.912525 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.912528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.912570 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.912645 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.913054 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43814: BlockingRecv error: recv got EOF from 127.0.0.1:43814 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 18:16:07.931943 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.930878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.23.65.125:45361)
0810 18:16:07.930980 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.930983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.931136 (+ 153us) server_negotiation.cc:407] Connection header received
0810 18:16:07.931264 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.931266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.931310 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.931367 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.931820 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43820: BlockingRecv error: recv got EOF from 127.0.0.1:43820 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 18:16:07.967043 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:07.965949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.23.65.125:45361)
0810 18:16:07.966105 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:07.966110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:07.966215 (+ 105us) server_negotiation.cc:407] Connection header received
0810 18:16:07.966371 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:07.966373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:07.966421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:07.966491 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:07.966944 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43830: BlockingRecv error: recv got EOF from 127.0.0.1:43830 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 18:16:08.034552 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:08.033186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43836 (local address 127.23.65.125:45361)
0810 18:16:08.033352 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:08.033357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:08.033477 (+ 120us) server_negotiation.cc:407] Connection header received
0810 18:16:08.033632 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:08.033634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:08.033838 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:08.033938 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:08.034441 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43836: BlockingRecv error: recv got EOF from 127.0.0.1:43836 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 18:16:08.166051 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:08.164866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.23.65.125:45361)
0810 18:16:08.165008 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:08.165013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:08.165186 (+ 173us) server_negotiation.cc:407] Connection header received
0810 18:16:08.165351 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:08.165353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:08.165405 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:08.165469 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:08.165945 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43846: BlockingRecv error: recv got EOF from 127.0.0.1:43846 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 18:16:08.425444 28711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:16:08.424380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.23.65.125:45361)
0810 18:16:08.424518 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:08.424522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:08.424541 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:16:08.424712 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:08.424714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:08.424773 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:08.424842 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:16:08.425334 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43858: BlockingRecv error: recv got EOF from 127.0.0.1:43858 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:16:08.717204 28888 catalog_manager.cc:1259] Loaded cluster ID: 5dfa4306107b46368266bf019a7d8eb4
I20260810 18:16:08.717298 28888 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562: loading cluster ID for follower catalog manager: success
I20260810 18:16:08.718961 28888 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562: acquiring CA information for follower catalog manager: success
I20260810 18:16:08.720129 28888 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7995c4b4985a427bb5dc6563d0a7f562: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:16:08.732334 28927 catalog_manager.cc:1259] Loaded cluster ID: 5dfa4306107b46368266bf019a7d8eb4
I20260810 18:16:08.732434 28927 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f: loading cluster ID for follower catalog manager: success
I20260810 18:16:08.734157 28927 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f: acquiring CA information for follower catalog manager: success
I20260810 18:16:08.735409 28927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4b9c86eb1c04a7e90ee19df2bd7e51f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:16:08.943164 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:08.939653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.23.65.125:45361)
0810 18:16:08.939902 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:08.939906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:08.939961 (+ 55us) server_negotiation.cc:407] Connection header received
0810 18:16:08.940081 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:08.940083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:08.940210 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:08.940334 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:08.940549 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.941069 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.941865 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.942075 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.942140 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:08.942397 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:08.942816 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:08.942907 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:08.942957 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 18:16:08.943005 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260810 18:16:08.949273 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:08.946035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.23.65.124:43659)
0810 18:16:08.946288 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:08.946291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:08.946304 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:16:08.946382 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:08.946384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:08.946496 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:08.946588 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:08.946736 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.947248 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.948075 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.948300 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.948372 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:08.948574 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:08.948954 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:08.949038 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:08.949062 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 18:16:08.949106 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260810 18:16:08.955618 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:08.952231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.23.65.125:45361)
0810 18:16:08.952343 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:08.952347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:08.952700 (+ 353us) server_negotiation.cc:407] Connection header received
0810 18:16:08.952872 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:08.952875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:08.952920 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:08.952980 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:08.953191 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.953670 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.954526 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.954762 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.954857 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:08.955171 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:08.955249 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:08.955321 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:08.955416 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 18:16:08.955465 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:16:08.956265 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:08.951921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.23.65.126:43177)
0810 18:16:08.952233 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:08.952237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:08.952253 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:16:08.952417 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:08.952421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:08.952572 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:08.952680 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:08.953028 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.953799 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.954872 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.955108 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.955201 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:08.955581 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:08.955897 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:08.955981 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:08.956079 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 18:16:08.956135 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":69,"threads_started":1}
I20260810 18:16:08.956543 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:16:08.952032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.23.65.124:43659)
0810 18:16:08.952129 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 18:16:08.952133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:16:08.952360 (+ 227us) server_negotiation.cc:407] Connection header received
0810 18:16:08.952536 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:16:08.952539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:16:08.952590 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:16:08.952657 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:16:08.952888 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.953633 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.954808 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:16:08.955094 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:16:08.955480 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:16:08.955567 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:16:08.955635 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:16:08.956354 (+ 719us) server_negotiation.cc:1035] Waiting for connection context
0810 18:16:08.956391 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 18:16:08.956437 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:16:08.958995 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 28465
I20260810 18:16:08.977101 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 28601
I20260810 18:16:08.988590 23813 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmz0U2F/build/debug/bin/kudu with pid 28716
2026-08-10T18:16:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3245 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6437 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (290207 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST